Sep 09 03:03:36.271454 ip-10-0-138-152 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 03:03:36.271466 ip-10-0-138-152 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 03:03:36.271473 ip-10-0-138-152 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 03:03:36.271705 ip-10-0-138-152 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 03:03:46.457262 ip-10-0-138-152 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 03:03:46.457281 ip-10-0-138-152 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6de482c6704f4ae79a1fb702ae77ca7e --
Sep 09 03:05:55.055971 ip-10-0-138-152 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 03:05:55.433978 ip-10-0-138-152 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:05:55.433978 ip-10-0-138-152 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 03:05:55.433978 ip-10-0-138-152 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:05:55.433978 ip-10-0-138-152 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 03:05:55.433978 ip-10-0-138-152 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:05:55.435026 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.434929    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 03:05:55.442425 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442409    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:55.442425 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442425    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442428    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442432    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442435    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442438    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442441    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442443    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442446    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442450    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442452    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442456    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442460    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442463    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442466    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442469    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442478    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442481    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442484    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442486    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442489    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:55.442494 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442491    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442494    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442496    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442499    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442502    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442505    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442508    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442511    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442514    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442516    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442519    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442521    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442524    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442526    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442531    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442535    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442538    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442540    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442543    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442546    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:55.443024 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442549    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442551    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442554    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442556    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442559    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442561    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442563    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442566    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442569    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442572    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442574    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442578    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442580    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442583    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442586    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442589    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442592    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442595    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442597    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442600    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:55.443583 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442602    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442605    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442607    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442610    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442613    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442615    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442618    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442620    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442631    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442633    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442636    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442638    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442641    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442644    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442646    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442648    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442651    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442653    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442656    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442658    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:55.444348 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442661    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442664    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442667    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442670    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442672    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442675    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442690    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442694    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442700    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442706    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442710    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442714    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442718    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442720    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442723    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442725    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442728    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.442730    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:55.445020 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445020    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445039    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445044    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445049    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445054    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445060    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445065    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445069    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445074    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445079    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445083    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445088    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445092    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445097    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445101    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445105    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445109    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445114    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445118    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445122    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:55.445812 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445126    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445131    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445135    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445139    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445148    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445153    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445157    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445161    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445166    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445170    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445174    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445178    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445183    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445187    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445198    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445204    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445209    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445214    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445218    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:55.446733 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445222    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445226    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445231    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445235    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445239    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445243    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445247    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445251    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445255    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445259    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445263    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445268    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445272    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445276    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445280    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445285    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445289    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445293    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445298    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:55.447570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445304    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445309    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445314    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445318    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445324    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445329    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445334    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445339    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445346    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445350    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445355    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445360    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445364    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445369    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445375    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445379    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445384    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445388    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445392    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445396    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:55.448345 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445400    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445404    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445409    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445413    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445417    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445421    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445425    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445429    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445433    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445437    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445442    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445446    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445450    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445455    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445459    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445463    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445467    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445471    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445475    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445479    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:55.448857 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.445484    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446115    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446126    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446131    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446135    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446139    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446144    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446148    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446152    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446156    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446161    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446165    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446169    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446173    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446177    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446183    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446188    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446194    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446199    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446203    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:55.449453 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446208    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446215    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446221    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446225    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446229    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446233    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446238    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446242    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446246    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446250    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446254    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446259    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446264    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446268    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446272    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446276    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446280    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446284    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446288    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:55.449993 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446292    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446296    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446300    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446304    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446308    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446312    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446316    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446321    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446325    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446329    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446333    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446338    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446343    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446347    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446351    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446356    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446360    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446364    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446368    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446372    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:55.450597 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446377    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446382    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446386    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446391    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446396    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446401    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446406    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446411    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446415    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446419    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446423    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446427    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446431    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446435    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446439    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446443    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446448    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446452    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446457    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446461    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:55.451245 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446466    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446470    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446474    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446478    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446484    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446489    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446493    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446497    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446502    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446505    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446510    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446514    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446518    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446522    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446526    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446530    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446537    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446543    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446548    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:55.451757 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446553    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.446557    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446661    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446735    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446771    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446780    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446786    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446791    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446797    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446804    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446809    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446814    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446819    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446825    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446830    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446835    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446839    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446844    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446854    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446860    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446867    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446872    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446877    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446883    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 03:05:55.452325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446888    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446894    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446899    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446904    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446909    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446914    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446918    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446923    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446928    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446932    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446939    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446944    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446948    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446953    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446958    2580 flags.go:64] FLAG: --enable-server="true"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446962    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446968    2580 flags.go:64] FLAG: --event-burst="100"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446973    2580 flags.go:64] FLAG: --event-qps="50"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446977    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446984    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.446989    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447003    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447008    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447012    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447017    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 09 03:05:55.452986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447021    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447029    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447034    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447039    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447044    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447049    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447054    2580 flags.go:64] FLAG: --feature-gates=""
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447060    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447065    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447070    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447075    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447080    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447085    2580 flags.go:64] FLAG: --help="false"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447089    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447095    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447099    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447104    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447110    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447115    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447120    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447125    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447129    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447134    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 03:05:55.453622 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447139    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447144    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447148    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447153    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447158    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447163    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447167    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447172    2580 flags.go:64] FLAG: --lock-file=""
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447176    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447181    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447189    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447198    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447204    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447208    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447213    2580 flags.go:64] FLAG: --logging-format="text"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447218    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447223    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447228    2580 flags.go:64] FLAG: --manifest-url=""
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447232    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447239    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447244    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447250    2580 flags.go:64] FLAG: --max-pods="110"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447255    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447259    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447264    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 03:05:55.454231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447268    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447273    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447278    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447283    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447294    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447299    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447304    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447309    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447314    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447323    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447328    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447333    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447337    2580 flags.go:64] FLAG: --port="10250"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447342    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447347    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d3a9af93a511adb3"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447352    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447357    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447362    2580 flags.go:64] FLAG: --register-node="true"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447368    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447374    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447380    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447385    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447390    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447402    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447407    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 03:05:55.454924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447412    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447417    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447422    2580 flags.go:64] FLAG: --runonce="false"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447426    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447431    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447436    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447440    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447445    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447450    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447455    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447459    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447464    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447469    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447473    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447478    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447483    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447487    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447492    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447500    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447505    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447509    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447516    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447520    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447525    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447529    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 03:05:55.455672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447536    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447541    2580 flags.go:64] FLAG: --v="2"
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447548    2580 flags.go:64] FLAG: --version="false"
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447555    2580 flags.go:64] FLAG: --vmodule=""
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447561    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.447566    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447755    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447764    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447769    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447773    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447778    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447782    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447786    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447790    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447794    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447799    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447804    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447809    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447813    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447817    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447822    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:55.456308 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447826    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447829    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447834    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447837    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447842    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447846    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447850    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447854    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447858    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447862    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447866    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447879    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447883    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447887    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447891    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447896    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447901    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447905    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447909    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447913    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:55.456850 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447917    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447921    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447925    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447930    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447934    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447938    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447944    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447950    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447956    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447961    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447965    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447970    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447974    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447979    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447983    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447988    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447991    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.447997    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:55.457369 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448004    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448008    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448012    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448016    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448020    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448027    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448031    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448035    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448039    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448043    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448049    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448054    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448058    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448062    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448066    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448070    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448074    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448078    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448082    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:55.457861 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448086    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448090    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448094    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448098    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448102    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448107    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448111    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448115    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448119    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448122    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448127    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448130    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448135    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448139    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448143    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448147    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448151    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448155    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448162    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448166    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:55.458327 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448170    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448174    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448178    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448182    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448187    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448192    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.448196    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.449035    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.456445    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.456459    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.457445    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.457452    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457508    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457514    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457518    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:55.458839 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457521    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457523    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457526    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457529    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457531    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457534    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457536    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457539    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457542    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457545    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457549    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457553    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457556    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457560    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457564    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457567    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457569    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457572    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457574    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:55.459218 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457577    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457580    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457582    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457585    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457587    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457590    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457592    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457596    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457599    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457601    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457604    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457607    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457609    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457612    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457614    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457617    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457619    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457622    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457624    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457627    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:55.459703 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457629    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457632    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457634    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457637    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457639    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457642    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457644    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457646    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457649    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457653    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457656    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457660    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457663    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457665    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457668    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457671    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457674    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457676    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457702    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:55.460205 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457705    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457709    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457712    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457716    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457718    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457721    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457724    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457726    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457729    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457731    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457735    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457737    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457740    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457742    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457745    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457747    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457750    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457752    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457755    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:55.460713 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457757    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457760    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457763    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457765    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457768    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457772    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457774    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457777    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457779    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457782    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457784    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457787    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457789    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457793    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457795    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457798    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457801    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457803    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:55.461176 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457805    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.457810    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457921    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457926    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457929    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457932    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457935    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457937    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457940    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457943    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457945    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457948    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457950    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457953    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457955    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:55.461608 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457958    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457960    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457962    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457965    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457968    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457970    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457973    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457975    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457978    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457980    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457984    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457988    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457991    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457994    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.457997    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458000    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458003    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458005    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458008    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:55.462003 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458010    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458013    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458015    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458018    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458021    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458023    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458025    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458028    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458031    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458035    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458038    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458040    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458043    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458045    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458048    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458050    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458054    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458056    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458059    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:55.462462 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458061    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458064    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458067    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458069    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458071    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458074    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458076    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458079    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458081    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458084    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458087    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458089    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458092    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458094    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458097    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458099    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458102    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458104    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458107    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458109    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:55.462942 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458111    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458114    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458116    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458118    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458121    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458123    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458126    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458128    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458130    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458133    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458135    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458138    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458140    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458143    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458147    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458150    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458153    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458156    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458158    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458161    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:55.463432 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458164    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458167    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458169    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458173    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458176    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458178    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458180    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:55.458183    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.458188    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.458291    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.459819    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.460641    2580 server.go:1019] "Starting client certificate rotation"
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.460741    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:05:55.463912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.460786    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:05:55.464234 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.463567    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:05:55.466255 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.466231    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:05:55.480881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.480864    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 09 03:05:55.484201 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.484184    2580 log.go:25] "Validated CRI v1 image API"
Sep 09 03:05:55.485111 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.485086    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 03:05:55.487693 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.487667    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:05:55.491044 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.491026    2580 fs.go:135] Filesystem UUIDs: map[1ff40888-2a15-4215-8055-5e751cc1a42b:/dev/nvme0n1p4 6304039d-08ae-46b9-bb60-6c79ebfd3ccf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 09 03:05:55.491106 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.491046    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 03:05:55.499077 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.498961    2580 manager.go:216] Machine: {Timestamp:2026-09-09 03:05:55.494672891 +0000 UTC m=+0.324883472 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3075192 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dc6f7a8ae449f7fca92a81ef9a2b7 SystemUUID:ec2dc6f7-a8ae-449f-7fca-92a81ef9a2b7 BootID:6de482c6-704f-4ae7-9a1f-b702ae77ca7e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0f:fb:8c:ad:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0f:fb:8c:ad:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:0a:ad:9d:e9:b0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 03:05:55.499077 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.499072    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 03:05:55.499180 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.499147    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 03:05:55.502421 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.502392    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 03:05:55.502563 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.502425    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-152.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 03:05:55.503232 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.503221    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 03:05:55.503263 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.503235    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 03:05:55.503263 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.503242    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:05:55.503916 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.503905    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:05:55.504613 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.504604    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 03:05:55.508129 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.508118    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:05:55.508236 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.508227    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 03:05:55.508265 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.508257    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 03:05:55.508292 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.508273    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 03:05:55.508292 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.508287    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 03:05:55.508351 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.508296    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 09 03:05:55.508351 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.508306    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 03:05:55.508741 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.508729    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:05:55.508741 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.508738    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xnnpz"
Sep 09 03:05:55.508849 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.508751    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:05:55.510934 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.510916    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 03:05:55.512905 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.512892    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 03:05:55.512958 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.512923    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 03:05:55.513587 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513576    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 03:05:55.513626 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513593    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 03:05:55.513626 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513599    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 03:05:55.513626 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513605    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 03:05:55.513626 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513613    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 03:05:55.513626 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513619    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 03:05:55.513784 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513673    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 03:05:55.513784 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513696    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 03:05:55.513784 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513704    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 03:05:55.513784 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513709    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 03:05:55.513784 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513717    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 03:05:55.513784 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.513725    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 03:05:55.515160 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.515147    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 03:05:55.515199 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.515163    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 03:05:55.517123 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.517104    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xnnpz"
Sep 09 03:05:55.518351 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.518328    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 03:05:55.518430 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.518382    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-152.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 03:05:55.518430 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.518383    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-152.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 03:05:55.519144 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.519129    2580 server.go:1268] "Started kubelet"
Sep 09 03:05:55.519254 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.519232    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 03:05:55.519830 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.519776    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 03:05:55.519931 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.519857    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 03:05:55.520106 ip-10-0-138-152 systemd[1]: Started Kubernetes Kubelet.
Sep 09 03:05:55.521752 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.521736    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 03:05:55.523807 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.523445    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 03:05:55.523969 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.523942    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 03:05:55.527024 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.527004    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 03:05:55.527446 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.527428    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 03:05:55.527628 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.527581    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:55.528219 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528187    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 03:05:55.528219 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528210    2580 factory.go:55] Registering systemd factory
Sep 09 03:05:55.528219 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528220    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 09 03:05:55.528363 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528348    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 03:05:55.528363 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528351    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 03:05:55.528429 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528379    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 03:05:55.528429 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528416    2580 factory.go:153] Registering CRI-O factory
Sep 09 03:05:55.528429 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528429    2580 factory.go:223] Registration of the crio container factory successfully
Sep 09 03:05:55.528521 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528448    2580 factory.go:103] Registering Raw factory
Sep 09 03:05:55.528521 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528463    2580 manager.go:1196] Started watching for new ooms in manager
Sep 09 03:05:55.528521 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528480    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 03:05:55.528521 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528487    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 03:05:55.528794 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.528782    2580 manager.go:349] Starting recovery of all containers
Sep 09 03:05:55.531663 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.531641    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:55.534529 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.534505    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-152.ec2.internal\" not found" node="ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.540442 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.540425    2580 manager.go:354] Recovery completed
Sep 09 03:05:55.544877 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.544864    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:55.547484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.547467    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:55.547560 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.547503    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:55.547560 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.547519    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:55.548211 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.548196    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 03:05:55.548265 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.548212    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 03:05:55.548265 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.548234    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:05:55.548337 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.548216    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:55.549053 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.549038    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:55.549107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.549063    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:55.549107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.549073    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:55.550346 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.550334    2580 policy_none.go:49] "None policy: Start"
Sep 09 03:05:55.550390 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.550350    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 03:05:55.550390 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.550359    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 03:05:55.552203 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.552192    2580 policy_none.go:47] "Start"
Sep 09 03:05:55.597548 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.597371    2580 manager.go:337] "Starting Device Plugin manager"
Sep 09 03:05:55.609867 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.597572    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 03:05:55.609867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.597586    2580 server.go:87] "Starting device plugin registration server"
Sep 09 03:05:55.609867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.597930    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 03:05:55.609867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.597945    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 03:05:55.609867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.598057    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 03:05:55.609867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.598140    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 03:05:55.609867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.598147    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 03:05:55.609867 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.598799    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 03:05:55.609867 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.598857    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:55.659240 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.659154    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 03:05:55.660575 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.660546    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 03:05:55.660710 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.660603    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 03:05:55.660710 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.660627    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 03:05:55.660710 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.660664    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 03:05:55.662943 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.662920    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:55.698984 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.698928    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:55.699968 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.699954    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:55.700035 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.699980    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:55.700035 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.699991    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:55.700035 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.700013    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.712149 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.712125    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.712228 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.712152    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-152.ec2.internal\": node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:55.740237 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.740205    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:55.761402 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.761178    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"]
Sep 09 03:05:55.761402 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.761265    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:55.767076 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.767055    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:55.767194 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.767088    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:55.767194 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.767099    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:55.769491 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.769474    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:55.769594 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.769575    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.769649 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.769614    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:55.771259 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.771241    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:55.771371 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.771274    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:55.771371 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.771279    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:55.771371 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.771306    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:55.771371 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.771315    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:55.771371 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.771288    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:55.773638 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.773623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.773709 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.773651    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:55.775269 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.775252    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:55.775381 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.775282    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:55.775381 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.775291    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:55.797482 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.797446    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-152.ec2.internal\" not found" node="ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.802080 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.802062    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-152.ec2.internal\" not found" node="ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.830074 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.830043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/684f3ba5af4d886cd161feb8e6005942-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"684f3ba5af4d886cd161feb8e6005942\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.830074 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.830074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/684f3ba5af4d886cd161feb8e6005942-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"684f3ba5af4d886cd161feb8e6005942\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.830273 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.830089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ce43aa27923e3198d26bf26fefbfe86-config\") pod \"kube-apiserver-proxy-ip-10-0-138-152.ec2.internal\" (UID: \"9ce43aa27923e3198d26bf26fefbfe86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.840578 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.840545    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:55.930938 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.930850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/684f3ba5af4d886cd161feb8e6005942-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"684f3ba5af4d886cd161feb8e6005942\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.930938 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.930891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/684f3ba5af4d886cd161feb8e6005942-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"684f3ba5af4d886cd161feb8e6005942\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.930938 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.930916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ce43aa27923e3198d26bf26fefbfe86-config\") pod \"kube-apiserver-proxy-ip-10-0-138-152.ec2.internal\" (UID: \"9ce43aa27923e3198d26bf26fefbfe86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.931152 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.930968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ce43aa27923e3198d26bf26fefbfe86-config\") pod \"kube-apiserver-proxy-ip-10-0-138-152.ec2.internal\" (UID: \"9ce43aa27923e3198d26bf26fefbfe86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.931152 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.930973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/684f3ba5af4d886cd161feb8e6005942-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"684f3ba5af4d886cd161feb8e6005942\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.931152 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:55.930974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/684f3ba5af4d886cd161feb8e6005942-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"684f3ba5af4d886cd161feb8e6005942\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:55.940982 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:55.940951    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:56.041718 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:56.041676    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:56.098894 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.098850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:56.105115 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.105094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:56.142314 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:56.142273    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:56.242875 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:56.242789    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:56.343373 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:56.343338    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:56.443925 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:56.443883    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:56.461060 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.461034    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 03:05:56.461206 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.461175    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 03:05:56.461256 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.461211    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 03:05:56.525786 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.525714    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 03:00:56 +0000 UTC" deadline="2028-02-14 03:58:14.3776541 +0000 UTC"
Sep 09 03:05:56.525786 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.525745    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12552h52m17.851912667s"
Sep 09 03:05:56.544210 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:56.544185    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:56.644819 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:56.644781    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:56.669570 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:56.669531    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684f3ba5af4d886cd161feb8e6005942.slice/crio-8ef0849fae07c5599f340c9d47d12000a2b5db7da8a030c954bf23ec525dbe27 WatchSource:0}: Error finding container 8ef0849fae07c5599f340c9d47d12000a2b5db7da8a030c954bf23ec525dbe27: Status 404 returned error can't find the container with id 8ef0849fae07c5599f340c9d47d12000a2b5db7da8a030c954bf23ec525dbe27
Sep 09 03:05:56.674363 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.674346    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:05:56.745509 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:56.745471    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Sep 09 03:05:56.760596 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.760576    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:56.828255 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.828229    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:56.840787 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.840767    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:05:56.842703 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.842668    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Sep 09 03:05:56.851732 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.851712    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:05:56.864924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:56.864905    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:58.814159 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.814128    2580 apiserver.go:52] "Watching apiserver"
Sep 09 03:05:58.824079 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.824056    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 03:05:58.826688 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.826657    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-jwqmr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal","openshift-multus/multus-additional-cni-plugins-72j6p","kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal","openshift-multus/multus-cd7wf","openshift-multus/network-metrics-daemon-6kfmb","openshift-network-diagnostics/network-check-target-gk5dp","openshift-network-operator/iptables-alerter-fvl79","openshift-ovn-kubernetes/ovnkube-node-8z4g8","kube-system/global-pull-secret-syncer-psdng","kube-system/konnectivity-agent-z2zb6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d","openshift-cluster-node-tuning-operator/tuned-4kpjc","openshift-dns/node-resolver-s6hwl"]
Sep 09 03:05:58.829104 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.829083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:05:58.829214 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:58.829164    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:05:58.831218 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.831197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.831633 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.831609    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 03:05:58.833156 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.833131    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:05:58.833309 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.833289    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 03:05:58.833513 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.833494    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.833611 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.833527    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xhrgh\""
Sep 09 03:05:58.835219 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.835191    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 03:05:58.835755 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.835392    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 03:05:58.835755 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.835441    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 03:05:58.835755 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.835591    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 03:05:58.835755 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.835630    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hmw4t\""
Sep 09 03:05:58.835755 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.835639    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 03:05:58.835755 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.835735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s6hwl"
Sep 09 03:05:58.837454 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.837436    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l9htr\""
Sep 09 03:05:58.837667 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.837653    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 03:05:58.837755 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.837692    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 03:05:58.837925 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.837905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.839777 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.839749    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l87bb\""
Sep 09 03:05:58.839777 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.839749    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 03:05:58.840049 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.840030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:05:58.840153 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:58.840099    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:05:58.842310 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.842463 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-sysctl-d\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.842541 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-lib-modules\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.842541 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-var-lib-kubelet\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.842541 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvr6r\" (UniqueName: \"kubernetes.io/projected/50cd46eb-3c10-43c8-b520-37f913a814cc-kube-api-access-jvr6r\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.842722 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8ab8583-facb-4d47-9197-83089e5468b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.842722 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-kubernetes\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.842722 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-sysctl-conf\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.842722 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-systemd\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.842722 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-sys\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.842722 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8ab8583-facb-4d47-9197-83089e5468b0-system-cni-dir\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.842722 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8ab8583-facb-4d47-9197-83089e5468b0-cnibin\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.843063 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8ab8583-facb-4d47-9197-83089e5468b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.843063 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8692cce-7d57-40d3-9994-7fdfc0adf178-dbus\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:05:58.843063 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:05:58.843063 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-run\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.843063 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-tuned\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.843063 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8ab8583-facb-4d47-9197-83089e5468b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.843063 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-host\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.843063 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.842966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50cd46eb-3c10-43c8-b520-37f913a814cc-tmp\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.843063 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.843012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8ab8583-facb-4d47-9197-83089e5468b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.843063 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.843057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbq4f\" (UniqueName: \"kubernetes.io/projected/a8ab8583-facb-4d47-9197-83089e5468b0-kube-api-access-vbq4f\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.843398 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.843092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8692cce-7d57-40d3-9994-7fdfc0adf178-kubelet-config\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:05:58.843398 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.843161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-modprobe-d\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.843398 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.843185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-sysconfig\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.844282 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.844263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 03:05:58.844748 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.844729    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 03:05:58.844848 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.844795    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sbgq6\""
Sep 09 03:05:58.844985 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.844967    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 03:05:58.845078 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.845062    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 03:05:58.845190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.845170    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 03:05:58.845294 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.845234    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:05:58.845462 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.845438    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 03:05:58.845611 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.845590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:05:58.845721 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:58.845687    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:05:58.847330 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.847312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fvl79"
Sep 09 03:05:58.849241 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.849175    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 03:05:58.849241 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.849193    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 03:05:58.849425 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.849404    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:05:58.849589 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.849558    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lchdn\""
Sep 09 03:05:58.849880 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.849859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jwqmr"
Sep 09 03:05:58.851621 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.851602    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 03:05:58.851840 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.851814    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 03:05:58.851936 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.851863    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5w9fz\""
Sep 09 03:05:58.851936 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.851886    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 03:05:58.852436 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.852420    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:05:58.854464 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.854259    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 03:05:58.854464 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.854282    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5qp5q\""
Sep 09 03:05:58.854464 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.854259    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 03:05:58.854689 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.854672    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:58.857020 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.856992    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 03:05:58.857110 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.857079    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 03:05:58.857208 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.857195    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 03:05:58.857746 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.857729    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-867vf\""
Sep 09 03:05:58.860038 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.860021    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hj6rj"
Sep 09 03:05:58.866717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.866700    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hj6rj"
Sep 09 03:05:58.933923 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.933891    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 03:05:58.943696 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-sysctl-conf\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.943855 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-sys\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.943855 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8ab8583-facb-4d47-9197-83089e5468b0-cnibin\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.943855 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-multus-socket-dir-parent\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.943855 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-run-netns\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.943855 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:58.943855 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-run\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.943855 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-tuned\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.944158 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-slash\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.944158 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-sysctl-conf\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.944158 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-run-systemd\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.944158 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4039f93-f06b-4c04-9870-ff92b01a66f4-host\") pod \"node-ca-jwqmr\" (UID: \"f4039f93-f06b-4c04-9870-ff92b01a66f4\") " pod="openshift-image-registry/node-ca-jwqmr"
Sep 09 03:05:58.944158 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-sys\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.944158 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da582df0-f11f-4ba7-9be7-b233515183ca-host-slash\") pod \"iptables-alerter-fvl79\" (UID: \"da582df0-f11f-4ba7-9be7-b233515183ca\") " pod="openshift-network-operator/iptables-alerter-fvl79"
Sep 09 03:05:58.944158 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.943999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6frhk\" (UniqueName: \"kubernetes.io/projected/d3d422be-7e29-4c78-9102-9975a8a3d2ad-kube-api-access-6frhk\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:58.944158 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8ab8583-facb-4d47-9197-83089e5468b0-cnibin\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.944158 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-run\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-host\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbq4f\" (UniqueName: \"kubernetes.io/projected/a8ab8583-facb-4d47-9197-83089e5468b0-kube-api-access-vbq4f\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-run-netns\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944262    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-var-lib-openvswitch\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-ovnkube-script-lib\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-host\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8692cce-7d57-40d3-9994-7fdfc0adf178-kubelet-config\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-sysctl-d\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944323    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8692cce-7d57-40d3-9994-7fdfc0adf178-kubelet-config\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-cnibin\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.944484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8ab8583-facb-4d47-9197-83089e5468b0-system-cni-dir\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-multus-conf-dir\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-sysctl-d\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-etc-kubernetes\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-ovn-node-metrics-cert\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8ab8583-facb-4d47-9197-83089e5468b0-system-cni-dir\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrsf\" (UniqueName: \"kubernetes.io/projected/f4039f93-f06b-4c04-9870-ff92b01a66f4-kube-api-access-rqrsf\") pod \"node-ca-jwqmr\" (UID: \"f4039f93-f06b-4c04-9870-ff92b01a66f4\") " pod="openshift-image-registry/node-ca-jwqmr"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d29b378-9477-47b3-9500-c47d907fbe49-hosts-file\") pod \"node-resolver-s6hwl\" (UID: \"0d29b378-9477-47b3-9500-c47d907fbe49\") " pod="openshift-dns/node-resolver-s6hwl"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br7vm\" (UniqueName: \"kubernetes.io/projected/da582df0-f11f-4ba7-9be7-b233515183ca-kube-api-access-br7vm\") pod \"iptables-alerter-fvl79\" (UID: \"da582df0-f11f-4ba7-9be7-b233515183ca\") " pod="openshift-network-operator/iptables-alerter-fvl79"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-device-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8692cce-7d57-40d3-9994-7fdfc0adf178-dbus\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-run-k8s-cni-cncf-io\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvfsv\" (UniqueName: \"kubernetes.io/projected/6d835db3-17d5-457e-a997-a2d7a61b516d-kube-api-access-zvfsv\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d29b378-9477-47b3-9500-c47d907fbe49-tmp-dir\") pod \"node-resolver-s6hwl\" (UID: \"0d29b378-9477-47b3-9500-c47d907fbe49\") " pod="openshift-dns/node-resolver-s6hwl"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt4g7\" (UniqueName: \"kubernetes.io/projected/0d29b378-9477-47b3-9500-c47d907fbe49-kube-api-access-lt4g7\") pod \"node-resolver-s6hwl\" (UID: \"0d29b378-9477-47b3-9500-c47d907fbe49\") " pod="openshift-dns/node-resolver-s6hwl"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50cd46eb-3c10-43c8-b520-37f913a814cc-tmp\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8692cce-7d57-40d3-9994-7fdfc0adf178-dbus\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:05:58.944990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-var-lib-cni-bin\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.944961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-kubelet\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-node-log\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-registration-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-lib-modules\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-var-lib-kubelet\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvr6r\" (UniqueName: \"kubernetes.io/projected/50cd46eb-3c10-43c8-b520-37f913a814cc-kube-api-access-jvr6r\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-etc-openvswitch\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-log-socket\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-env-overrides\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-var-lib-kubelet\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-system-cni-dir\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-cni-bin\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4896cb96-b860-4f7c-aaa8-2abb525ed851-konnectivity-ca\") pod \"konnectivity-agent-z2zb6\" (UID: \"4896cb96-b860-4f7c-aaa8-2abb525ed851\") " pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-sys-fs\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8ab8583-facb-4d47-9197-83089e5468b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.945692 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmjp\" (UniqueName: \"kubernetes.io/projected/ee670977-001f-4a64-8c78-4245ad3befe8-kube-api-access-slmjp\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-ovnkube-config\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da582df0-f11f-4ba7-9be7-b233515183ca-iptables-alerter-script\") pod \"iptables-alerter-fvl79\" (UID: \"da582df0-f11f-4ba7-9be7-b233515183ca\") " pod="openshift-network-operator/iptables-alerter-fvl79"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-socket-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8ab8583-facb-4d47-9197-83089e5468b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-multus-cni-dir\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee670977-001f-4a64-8c78-4245ad3befe8-cni-binary-copy\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-hostroot\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hwsv\" (UniqueName: \"kubernetes.io/projected/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-kube-api-access-7hwsv\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tlgd\" (UniqueName: \"kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd\") pod \"network-check-target-gk5dp\" (UID: \"71a3d2a6-efff-42dc-a3a6-19dbba96041a\") " pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-modprobe-d\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-var-lib-kubelet\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee670977-001f-4a64-8c78-4245ad3befe8-multus-daemon-config\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8ab8583-facb-4d47-9197-83089e5468b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.946171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-run-ovn\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.945982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4039f93-f06b-4c04-9870-ff92b01a66f4-serviceca\") pod \"node-ca-jwqmr\" (UID: \"f4039f93-f06b-4c04-9870-ff92b01a66f4\") " pod="openshift-image-registry/node-ca-jwqmr"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-kubernetes\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-systemd\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8ab8583-facb-4d47-9197-83089e5468b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-lib-modules\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-run-multus-certs\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-modprobe-d\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-var-lib-cni-multus\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-cni-netd\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-systemd\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8ab8583-facb-4d47-9197-83089e5468b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4896cb96-b860-4f7c-aaa8-2abb525ed851-agent-certs\") pod \"konnectivity-agent-z2zb6\" (UID: \"4896cb96-b860-4f7c-aaa8-2abb525ed851\") " pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-sysconfig\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8ab8583-facb-4d47-9197-83089e5468b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.946705 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:58.946477    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:58.947219 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-systemd-units\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.947219 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:58.946615    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret podName:d8692cce-7d57-40d3-9994-7fdfc0adf178 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:59.446563322 +0000 UTC m=+2.972878058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret") pod "global-pull-secret-syncer-psdng" (UID: "d8692cce-7d57-40d3-9994-7fdfc0adf178") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:58.947219 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-run-openvswitch\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:58.947219 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8ab8583-facb-4d47-9197-83089e5468b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.947219 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-kubernetes\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.947219 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.946930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-sysconfig\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.947393 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.947382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8ab8583-facb-4d47-9197-83089e5468b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.947990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.947971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50cd46eb-3c10-43c8-b520-37f913a814cc-tmp\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.948147 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.948027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50cd46eb-3c10-43c8-b520-37f913a814cc-etc-tuned\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.951714 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.951680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbq4f\" (UniqueName: \"kubernetes.io/projected/a8ab8583-facb-4d47-9197-83089e5468b0-kube-api-access-vbq4f\") pod \"multus-additional-cni-plugins-72j6p\" (UID: \"a8ab8583-facb-4d47-9197-83089e5468b0\") " pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:58.951891 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.951866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvr6r\" (UniqueName: \"kubernetes.io/projected/50cd46eb-3c10-43c8-b520-37f913a814cc-kube-api-access-jvr6r\") pod \"tuned-4kpjc\" (UID: \"50cd46eb-3c10-43c8-b520-37f913a814cc\") " pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:58.969303 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:58.969257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal" event={"ID":"684f3ba5af4d886cd161feb8e6005942","Type":"ContainerStarted","Data":"8ef0849fae07c5599f340c9d47d12000a2b5db7da8a030c954bf23ec525dbe27"}
Sep 09 03:05:59.047634 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-multus-conf-dir\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.047634 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-etc-kubernetes\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.047868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-ovn-node-metrics-cert\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.047868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrsf\" (UniqueName: \"kubernetes.io/projected/f4039f93-f06b-4c04-9870-ff92b01a66f4-kube-api-access-rqrsf\") pod \"node-ca-jwqmr\" (UID: \"f4039f93-f06b-4c04-9870-ff92b01a66f4\") " pod="openshift-image-registry/node-ca-jwqmr"
Sep 09 03:05:59.047868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d29b378-9477-47b3-9500-c47d907fbe49-hosts-file\") pod \"node-resolver-s6hwl\" (UID: \"0d29b378-9477-47b3-9500-c47d907fbe49\") " pod="openshift-dns/node-resolver-s6hwl"
Sep 09 03:05:59.047868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-multus-conf-dir\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.047868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br7vm\" (UniqueName: \"kubernetes.io/projected/da582df0-f11f-4ba7-9be7-b233515183ca-kube-api-access-br7vm\") pod \"iptables-alerter-fvl79\" (UID: \"da582df0-f11f-4ba7-9be7-b233515183ca\") " pod="openshift-network-operator/iptables-alerter-fvl79"
Sep 09 03:05:59.047868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-etc-kubernetes\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.047868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-device-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.047868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-run-k8s-cni-cncf-io\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.047868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvfsv\" (UniqueName: \"kubernetes.io/projected/6d835db3-17d5-457e-a997-a2d7a61b516d-kube-api-access-zvfsv\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:05:59.047868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d29b378-9477-47b3-9500-c47d907fbe49-hosts-file\") pod \"node-resolver-s6hwl\" (UID: \"0d29b378-9477-47b3-9500-c47d907fbe49\") " pod="openshift-dns/node-resolver-s6hwl"
Sep 09 03:05:59.047868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d29b378-9477-47b3-9500-c47d907fbe49-tmp-dir\") pod \"node-resolver-s6hwl\" (UID: \"0d29b378-9477-47b3-9500-c47d907fbe49\") " pod="openshift-dns/node-resolver-s6hwl"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-run-k8s-cni-cncf-io\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-device-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.047977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt4g7\" (UniqueName: \"kubernetes.io/projected/0d29b378-9477-47b3-9500-c47d907fbe49-kube-api-access-lt4g7\") pod \"node-resolver-s6hwl\" (UID: \"0d29b378-9477-47b3-9500-c47d907fbe49\") " pod="openshift-dns/node-resolver-s6hwl"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-var-lib-cni-bin\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-kubelet\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d29b378-9477-47b3-9500-c47d907fbe49-tmp-dir\") pod \"node-resolver-s6hwl\" (UID: \"0d29b378-9477-47b3-9500-c47d907fbe49\") " pod="openshift-dns/node-resolver-s6hwl"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-node-log\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-node-log\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-kubelet\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-var-lib-cni-bin\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-registration-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-registration-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-etc-openvswitch\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-log-socket\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-etc-openvswitch\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-env-overrides\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.048359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048367    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-log-socket\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-system-cni-dir\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-cni-bin\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4896cb96-b860-4f7c-aaa8-2abb525ed851-konnectivity-ca\") pod \"konnectivity-agent-z2zb6\" (UID: \"4896cb96-b860-4f7c-aaa8-2abb525ed851\") " pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-sys-fs\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmjp\" (UniqueName: \"kubernetes.io/projected/ee670977-001f-4a64-8c78-4245ad3befe8-kube-api-access-slmjp\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-system-cni-dir\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-ovnkube-config\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da582df0-f11f-4ba7-9be7-b233515183ca-iptables-alerter-script\") pod \"iptables-alerter-fvl79\" (UID: \"da582df0-f11f-4ba7-9be7-b233515183ca\") " pod="openshift-network-operator/iptables-alerter-fvl79"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-socket-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-multus-cni-dir\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee670977-001f-4a64-8c78-4245ad3befe8-cni-binary-copy\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-hostroot\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-cni-bin\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hwsv\" (UniqueName: \"kubernetes.io/projected/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-kube-api-access-7hwsv\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.049306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlgd\" (UniqueName: \"kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd\") pod \"network-check-target-gk5dp\" (UID: \"71a3d2a6-efff-42dc-a3a6-19dbba96041a\") " pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-var-lib-kubelet\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-env-overrides\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee670977-001f-4a64-8c78-4245ad3befe8-multus-daemon-config\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-run-ovn\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4039f93-f06b-4c04-9870-ff92b01a66f4-serviceca\") pod \"node-ca-jwqmr\" (UID: \"f4039f93-f06b-4c04-9870-ff92b01a66f4\") " pod="openshift-image-registry/node-ca-jwqmr"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-run-multus-certs\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-multus-cni-dir\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-var-lib-kubelet\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-var-lib-cni-multus\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049025    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-run-multus-certs\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4896cb96-b860-4f7c-aaa8-2abb525ed851-konnectivity-ca\") pod \"konnectivity-agent-z2zb6\" (UID: \"4896cb96-b860-4f7c-aaa8-2abb525ed851\") " pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.049057    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-hostroot\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.048918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-var-lib-cni-multus\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.049123    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs podName:6d835db3-17d5-457e-a997-a2d7a61b516d nodeName:}" failed. No retries permitted until 2026-09-09 03:05:59.549107157 +0000 UTC m=+3.075421879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs") pod "network-metrics-daemon-6kfmb" (UID: "6d835db3-17d5-457e-a997-a2d7a61b516d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:59.050113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-sys-fs\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-run-ovn\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-socket-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-ovnkube-config\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4039f93-f06b-4c04-9870-ff92b01a66f4-serviceca\") pod \"node-ca-jwqmr\" (UID: \"f4039f93-f06b-4c04-9870-ff92b01a66f4\") " pod="openshift-image-registry/node-ca-jwqmr"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-cni-netd\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4896cb96-b860-4f7c-aaa8-2abb525ed851-agent-certs\") pod \"konnectivity-agent-z2zb6\" (UID: \"4896cb96-b860-4f7c-aaa8-2abb525ed851\") " pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-cni-netd\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-systemd-units\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-run-openvswitch\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-systemd-units\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-multus-socket-dir-parent\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-run-openvswitch\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-run-netns\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-multus-socket-dir-parent\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.050847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-slash\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-host-run-netns\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-run-systemd\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4039f93-f06b-4c04-9870-ff92b01a66f4-host\") pod \"node-ca-jwqmr\" (UID: \"f4039f93-f06b-4c04-9870-ff92b01a66f4\") " pod="openshift-image-registry/node-ca-jwqmr"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-run-systemd\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da582df0-f11f-4ba7-9be7-b233515183ca-host-slash\") pod \"iptables-alerter-fvl79\" (UID: \"da582df0-f11f-4ba7-9be7-b233515183ca\") " pod="openshift-network-operator/iptables-alerter-fvl79"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da582df0-f11f-4ba7-9be7-b233515183ca-iptables-alerter-script\") pod \"iptables-alerter-fvl79\" (UID: \"da582df0-f11f-4ba7-9be7-b233515183ca\") " pod="openshift-network-operator/iptables-alerter-fvl79"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6frhk\" (UniqueName: \"kubernetes.io/projected/d3d422be-7e29-4c78-9102-9975a8a3d2ad-kube-api-access-6frhk\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-run-netns\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4039f93-f06b-4c04-9870-ff92b01a66f4-host\") pod \"node-ca-jwqmr\" (UID: \"f4039f93-f06b-4c04-9870-ff92b01a66f4\") " pod="openshift-image-registry/node-ca-jwqmr"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-var-lib-openvswitch\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3d422be-7e29-4c78-9102-9975a8a3d2ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049798    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-slash\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-ovnkube-script-lib\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-run-netns\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-cnibin\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da582df0-f11f-4ba7-9be7-b233515183ca-host-slash\") pod \"iptables-alerter-fvl79\" (UID: \"da582df0-f11f-4ba7-9be7-b233515183ca\") " pod="openshift-network-operator/iptables-alerter-fvl79"
Sep 09 03:05:59.051481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-var-lib-openvswitch\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.052132 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.052132 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.049935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee670977-001f-4a64-8c78-4245ad3befe8-cnibin\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.052132 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.050069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee670977-001f-4a64-8c78-4245ad3befe8-multus-daemon-config\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.052132 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.050075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee670977-001f-4a64-8c78-4245ad3befe8-cni-binary-copy\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.052132 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.050264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-ovnkube-script-lib\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.052132 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.050871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-ovn-node-metrics-cert\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.052429 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.052349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4896cb96-b860-4f7c-aaa8-2abb525ed851-agent-certs\") pod \"konnectivity-agent-z2zb6\" (UID: \"4896cb96-b860-4f7c-aaa8-2abb525ed851\") " pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:05:59.056928 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.056843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br7vm\" (UniqueName: \"kubernetes.io/projected/da582df0-f11f-4ba7-9be7-b233515183ca-kube-api-access-br7vm\") pod \"iptables-alerter-fvl79\" (UID: \"da582df0-f11f-4ba7-9be7-b233515183ca\") " pod="openshift-network-operator/iptables-alerter-fvl79"
Sep 09 03:05:59.057035 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.056941    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:05:59.057035 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.056960    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:05:59.057035 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.056972    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8tlgd for pod openshift-network-diagnostics/network-check-target-gk5dp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:59.057200 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.057047    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd podName:71a3d2a6-efff-42dc-a3a6-19dbba96041a nodeName:}" failed. No retries permitted until 2026-09-09 03:05:59.557027925 +0000 UTC m=+3.083342659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8tlgd" (UniqueName: "kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd") pod "network-check-target-gk5dp" (UID: "71a3d2a6-efff-42dc-a3a6-19dbba96041a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:59.057200 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.057078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvfsv\" (UniqueName: \"kubernetes.io/projected/6d835db3-17d5-457e-a997-a2d7a61b516d-kube-api-access-zvfsv\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:05:59.057200 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.057086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmjp\" (UniqueName: \"kubernetes.io/projected/ee670977-001f-4a64-8c78-4245ad3befe8-kube-api-access-slmjp\") pod \"multus-cd7wf\" (UID: \"ee670977-001f-4a64-8c78-4245ad3befe8\") " pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.057369 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.057285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrsf\" (UniqueName: \"kubernetes.io/projected/f4039f93-f06b-4c04-9870-ff92b01a66f4-kube-api-access-rqrsf\") pod \"node-ca-jwqmr\" (UID: \"f4039f93-f06b-4c04-9870-ff92b01a66f4\") " pod="openshift-image-registry/node-ca-jwqmr"
Sep 09 03:05:59.057668 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.057646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt4g7\" (UniqueName: \"kubernetes.io/projected/0d29b378-9477-47b3-9500-c47d907fbe49-kube-api-access-lt4g7\") pod \"node-resolver-s6hwl\" (UID: \"0d29b378-9477-47b3-9500-c47d907fbe49\") " pod="openshift-dns/node-resolver-s6hwl"
Sep 09 03:05:59.058820 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.058797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hwsv\" (UniqueName: \"kubernetes.io/projected/bb29a572-1fbe-4fba-bc0b-36fb778eaf24-kube-api-access-7hwsv\") pod \"ovnkube-node-8z4g8\" (UID: \"bb29a572-1fbe-4fba-bc0b-36fb778eaf24\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.059170 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.059148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6frhk\" (UniqueName: \"kubernetes.io/projected/d3d422be-7e29-4c78-9102-9975a8a3d2ad-kube-api-access-6frhk\") pod \"aws-ebs-csi-driver-node-zsl7d\" (UID: \"d3d422be-7e29-4c78-9102-9975a8a3d2ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.143514 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.143473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4kpjc"
Sep 09 03:05:59.152121 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.152092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-72j6p"
Sep 09 03:05:59.161145 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.161119    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s6hwl"
Sep 09 03:05:59.162149 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.162132    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:59.165992 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.165968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cd7wf"
Sep 09 03:05:59.170716 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.170697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:05:59.178384 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.178363    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fvl79"
Sep 09 03:05:59.183173 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.183152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jwqmr"
Sep 09 03:05:59.189970 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.189939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:05:59.193527 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.193511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d"
Sep 09 03:05:59.216749 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:59.216718    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d29b378_9477_47b3_9500_c47d907fbe49.slice/crio-bf21258886d1f6c9211e64a1b04b2e968cf96806d610a2fb23bc7f3aa639e8aa WatchSource:0}: Error finding container bf21258886d1f6c9211e64a1b04b2e968cf96806d610a2fb23bc7f3aa639e8aa: Status 404 returned error can't find the container with id bf21258886d1f6c9211e64a1b04b2e968cf96806d610a2fb23bc7f3aa639e8aa
Sep 09 03:05:59.218508 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.218057    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:59.222771 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:59.222744    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cd46eb_3c10_43c8_b520_37f913a814cc.slice/crio-e5fd7865d97641b8645be82ebd77d0bf004d87794a98f0ad9a4475536800d064 WatchSource:0}: Error finding container e5fd7865d97641b8645be82ebd77d0bf004d87794a98f0ad9a4475536800d064: Status 404 returned error can't find the container with id e5fd7865d97641b8645be82ebd77d0bf004d87794a98f0ad9a4475536800d064
Sep 09 03:05:59.223347 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:59.223319    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4896cb96_b860_4f7c_aaa8_2abb525ed851.slice/crio-9b8f86b1384cb921bbf5967db9b1ca3be43f7dba379a40bc54a144d55d533edc WatchSource:0}: Error finding container 9b8f86b1384cb921bbf5967db9b1ca3be43f7dba379a40bc54a144d55d533edc: Status 404 returned error can't find the container with id 9b8f86b1384cb921bbf5967db9b1ca3be43f7dba379a40bc54a144d55d533edc
Sep 09 03:05:59.224839 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:59.224409    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda582df0_f11f_4ba7_9be7_b233515183ca.slice/crio-cbdcfde7f5aa81ccaf8abb84b4b8aa6e36e70367730077e3cbc1a4fdfa845813 WatchSource:0}: Error finding container cbdcfde7f5aa81ccaf8abb84b4b8aa6e36e70367730077e3cbc1a4fdfa845813: Status 404 returned error can't find the container with id cbdcfde7f5aa81ccaf8abb84b4b8aa6e36e70367730077e3cbc1a4fdfa845813
Sep 09 03:05:59.226571 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:59.225234    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee670977_001f_4a64_8c78_4245ad3befe8.slice/crio-44f3b0cdc27bbd4a40e8610165cf045af0fdd8de23b3c3fabc07548aad23247e WatchSource:0}: Error finding container 44f3b0cdc27bbd4a40e8610165cf045af0fdd8de23b3c3fabc07548aad23247e: Status 404 returned error can't find the container with id 44f3b0cdc27bbd4a40e8610165cf045af0fdd8de23b3c3fabc07548aad23247e
Sep 09 03:05:59.270658 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:05:59.270634    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce43aa27923e3198d26bf26fefbfe86.slice/crio-62a2057123be251933dae89f10235e864028ba73126aa8a9025c9b8696ba314a WatchSource:0}: Error finding container 62a2057123be251933dae89f10235e864028ba73126aa8a9025c9b8696ba314a: Status 404 returned error can't find the container with id 62a2057123be251933dae89f10235e864028ba73126aa8a9025c9b8696ba314a
Sep 09 03:05:59.452791 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.452757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:05:59.452920 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.452899    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:59.452972 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.452962    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret podName:d8692cce-7d57-40d3-9994-7fdfc0adf178 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:00.452946765 +0000 UTC m=+3.979261481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret") pod "global-pull-secret-syncer-psdng" (UID: "d8692cce-7d57-40d3-9994-7fdfc0adf178") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:59.554116 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.554078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:05:59.554263 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.554235    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:59.554307 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.554297    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs podName:6d835db3-17d5-457e-a997-a2d7a61b516d nodeName:}" failed. No retries permitted until 2026-09-09 03:06:00.554283831 +0000 UTC m=+4.080598548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs") pod "network-metrics-daemon-6kfmb" (UID: "6d835db3-17d5-457e-a997-a2d7a61b516d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:59.655260 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.655223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlgd\" (UniqueName: \"kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd\") pod \"network-check-target-gk5dp\" (UID: \"71a3d2a6-efff-42dc-a3a6-19dbba96041a\") " pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:05:59.655412 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.655364    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:05:59.655412 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.655386    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:05:59.655412 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.655397    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8tlgd for pod openshift-network-diagnostics/network-check-target-gk5dp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:59.655562 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:05:59.655462    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd podName:71a3d2a6-efff-42dc-a3a6-19dbba96041a nodeName:}" failed. No retries permitted until 2026-09-09 03:06:00.65543541 +0000 UTC m=+4.181750147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tlgd" (UniqueName: "kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd") pod "network-check-target-gk5dp" (UID: "71a3d2a6-efff-42dc-a3a6-19dbba96041a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:59.869416 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.869320    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:00:58 +0000 UTC" deadline="2028-03-10 14:00:54.312347128 +0000 UTC"
Sep 09 03:05:59.869416 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.869358    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13162h54m54.442993409s"
Sep 09 03:05:59.975288 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.975248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4kpjc" event={"ID":"50cd46eb-3c10-43c8-b520-37f913a814cc","Type":"ContainerStarted","Data":"e5fd7865d97641b8645be82ebd77d0bf004d87794a98f0ad9a4475536800d064"}
Sep 09 03:05:59.982650 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.982603    2580 generic.go:356] "Generic (PLEG): container finished" podID="684f3ba5af4d886cd161feb8e6005942" containerID="77de9b9375cee411c95a2c69ba3b3586f0888d71e38d3a10edd46401fb362b3f" exitCode=0
Sep 09 03:05:59.982806 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.982712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal" event={"ID":"684f3ba5af4d886cd161feb8e6005942","Type":"ContainerDied","Data":"77de9b9375cee411c95a2c69ba3b3586f0888d71e38d3a10edd46401fb362b3f"}
Sep 09 03:05:59.988807 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:05:59.988776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8" event={"ID":"bb29a572-1fbe-4fba-bc0b-36fb778eaf24","Type":"ContainerStarted","Data":"5b82e2cf3d5826ada30956fec7f2042c3621d2d18e26b0d57d5bd8b73e3ee37d"}
Sep 09 03:06:00.010615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.009083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jwqmr" event={"ID":"f4039f93-f06b-4c04-9870-ff92b01a66f4","Type":"ContainerStarted","Data":"023bf8fc01623996b81b6f8d9f8bada90f31c552f0904d1eb520fc8273420396"}
Sep 09 03:06:00.020126 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.020088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d" event={"ID":"d3d422be-7e29-4c78-9102-9975a8a3d2ad","Type":"ContainerStarted","Data":"49be96a7fc1342246c51b71a626f51985fccebfc7e72bfbf25a00457df910baa"}
Sep 09 03:06:00.029004 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.028955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fvl79" event={"ID":"da582df0-f11f-4ba7-9be7-b233515183ca","Type":"ContainerStarted","Data":"cbdcfde7f5aa81ccaf8abb84b4b8aa6e36e70367730077e3cbc1a4fdfa845813"}
Sep 09 03:06:00.053716 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.053674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s6hwl" event={"ID":"0d29b378-9477-47b3-9500-c47d907fbe49","Type":"ContainerStarted","Data":"bf21258886d1f6c9211e64a1b04b2e968cf96806d610a2fb23bc7f3aa639e8aa"}
Sep 09 03:06:00.067450 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.067405    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal" event={"ID":"9ce43aa27923e3198d26bf26fefbfe86","Type":"ContainerStarted","Data":"62a2057123be251933dae89f10235e864028ba73126aa8a9025c9b8696ba314a"}
Sep 09 03:06:00.076535 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.076496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72j6p" event={"ID":"a8ab8583-facb-4d47-9197-83089e5468b0","Type":"ContainerStarted","Data":"1400c14944ec28ab074a62b42b119d0d80b5fc17a7684b7404b8906b9d8e607e"}
Sep 09 03:06:00.090849 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.090808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cd7wf" event={"ID":"ee670977-001f-4a64-8c78-4245ad3befe8","Type":"ContainerStarted","Data":"44f3b0cdc27bbd4a40e8610165cf045af0fdd8de23b3c3fabc07548aad23247e"}
Sep 09 03:06:00.107287 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.107162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z2zb6" event={"ID":"4896cb96-b860-4f7c-aaa8-2abb525ed851","Type":"ContainerStarted","Data":"9b8f86b1384cb921bbf5967db9b1ca3be43f7dba379a40bc54a144d55d533edc"}
Sep 09 03:06:00.462404 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.462066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:00.462652 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:00.462413    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:00.462652 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:00.462498    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret podName:d8692cce-7d57-40d3-9994-7fdfc0adf178 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:02.46247599 +0000 UTC m=+5.988790731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret") pod "global-pull-secret-syncer-psdng" (UID: "d8692cce-7d57-40d3-9994-7fdfc0adf178") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:00.563112 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.563070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:00.563315 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:00.563296    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:00.563440 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:00.563380    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs podName:6d835db3-17d5-457e-a997-a2d7a61b516d nodeName:}" failed. No retries permitted until 2026-09-09 03:06:02.563359886 +0000 UTC m=+6.089674607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs") pod "network-metrics-daemon-6kfmb" (UID: "6d835db3-17d5-457e-a997-a2d7a61b516d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:00.667632 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.667553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlgd\" (UniqueName: \"kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd\") pod \"network-check-target-gk5dp\" (UID: \"71a3d2a6-efff-42dc-a3a6-19dbba96041a\") " pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:00.667780 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:00.667716    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:06:00.667780 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:00.667731    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:06:00.667780 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:00.667741    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8tlgd for pod openshift-network-diagnostics/network-check-target-gk5dp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:00.667941 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:00.667792    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd podName:71a3d2a6-efff-42dc-a3a6-19dbba96041a nodeName:}" failed. No retries permitted until 2026-09-09 03:06:02.667776366 +0000 UTC m=+6.194091083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tlgd" (UniqueName: "kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd") pod "network-check-target-gk5dp" (UID: "71a3d2a6-efff-42dc-a3a6-19dbba96041a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:00.869752 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.869662    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:00:58 +0000 UTC" deadline="2028-04-15 20:58:33.226495481 +0000 UTC"
Sep 09 03:06:00.869752 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.869720    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14033h52m32.356779191s"
Sep 09 03:06:00.967953 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.967107    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:00.967953 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:00.967244    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:00.967953 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.967658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:00.967953 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:00.967753    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:00.967953 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:00.967816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:00.967953 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:00.967888    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:01.122526 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:01.122443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal" event={"ID":"684f3ba5af4d886cd161feb8e6005942","Type":"ContainerStarted","Data":"11bb7b84b5bb2c0190cf02c337711cc2bb9c67c776e9ab61d46df77bc61f575e"}
Sep 09 03:06:02.482761 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:02.482725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:02.483215 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:02.482869    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:02.483215 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:02.482941    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret podName:d8692cce-7d57-40d3-9994-7fdfc0adf178 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:06.482922587 +0000 UTC m=+10.009237305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret") pod "global-pull-secret-syncer-psdng" (UID: "d8692cce-7d57-40d3-9994-7fdfc0adf178") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:02.584504 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:02.583812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:02.584504 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:02.584020    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:02.584504 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:02.584084    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs podName:6d835db3-17d5-457e-a997-a2d7a61b516d nodeName:}" failed. No retries permitted until 2026-09-09 03:06:06.584065126 +0000 UTC m=+10.110379843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs") pod "network-metrics-daemon-6kfmb" (UID: "6d835db3-17d5-457e-a997-a2d7a61b516d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:02.685161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:02.684571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlgd\" (UniqueName: \"kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd\") pod \"network-check-target-gk5dp\" (UID: \"71a3d2a6-efff-42dc-a3a6-19dbba96041a\") " pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:02.685161 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:02.684710    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:06:02.685161 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:02.684735    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:06:02.685161 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:02.684747    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8tlgd for pod openshift-network-diagnostics/network-check-target-gk5dp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:02.685161 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:02.684813    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd podName:71a3d2a6-efff-42dc-a3a6-19dbba96041a nodeName:}" failed. No retries permitted until 2026-09-09 03:06:06.684794716 +0000 UTC m=+10.211109433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tlgd" (UniqueName: "kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd") pod "network-check-target-gk5dp" (UID: "71a3d2a6-efff-42dc-a3a6-19dbba96041a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:02.966374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:02.965677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:02.966374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:02.965702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:02.966374 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:02.965814    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:02.966374 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:02.966220    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:02.966374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:02.966268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:02.966374 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:02.966336    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:04.965791 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:04.965755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:04.966274 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:04.965799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:04.966274 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:04.965755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:04.966274 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:04.965915    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:04.966274 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:04.965988    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:04.966274 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:04.966064    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:06.516339 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:06.516300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:06.516728 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:06.516466    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:06.516728 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:06.516542    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret podName:d8692cce-7d57-40d3-9994-7fdfc0adf178 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:14.51652 +0000 UTC m=+18.042834736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret") pod "global-pull-secret-syncer-psdng" (UID: "d8692cce-7d57-40d3-9994-7fdfc0adf178") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:06.616714 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:06.616671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:06.616886 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:06.616869    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:06.616961 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:06.616945    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs podName:6d835db3-17d5-457e-a997-a2d7a61b516d nodeName:}" failed. No retries permitted until 2026-09-09 03:06:14.616925272 +0000 UTC m=+18.143240003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs") pod "network-metrics-daemon-6kfmb" (UID: "6d835db3-17d5-457e-a997-a2d7a61b516d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:06.717794 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:06.717714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlgd\" (UniqueName: \"kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd\") pod \"network-check-target-gk5dp\" (UID: \"71a3d2a6-efff-42dc-a3a6-19dbba96041a\") " pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:06.717957 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:06.717898    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:06:06.717957 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:06.717926    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:06:06.717957 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:06.717941    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8tlgd for pod openshift-network-diagnostics/network-check-target-gk5dp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:06.718070 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:06.718003    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd podName:71a3d2a6-efff-42dc-a3a6-19dbba96041a nodeName:}" failed. No retries permitted until 2026-09-09 03:06:14.717982257 +0000 UTC m=+18.244296994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tlgd" (UniqueName: "kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd") pod "network-check-target-gk5dp" (UID: "71a3d2a6-efff-42dc-a3a6-19dbba96041a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:06.966733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:06.966692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:06.966733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:06.966726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:06.966994 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:06.966692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:06.966994 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:06.966814    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:06.966994 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:06.966963    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:06.967193 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:06.967043    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:08.964814 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:08.964772    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:08.965289 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:08.964778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:08.965289 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:08.964927    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:08.965289 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:08.964778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:08.965289 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:08.964988    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:08.965289 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:08.965051    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:10.964834 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:10.964808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:10.964834 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:10.964822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:10.965224 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:10.964904    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:10.965224 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:10.964997    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:10.965224 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:10.965029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:10.965224 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:10.965090    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:12.965141 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:12.965094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:12.965524 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:12.965159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:12.965524 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:12.965169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:12.965524 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:12.965252    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:12.965524 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:12.965318    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:12.965524 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:12.965393    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:14.572078 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:14.572048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:14.572493 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:14.572142    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:14.572493 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:14.572189    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret podName:d8692cce-7d57-40d3-9994-7fdfc0adf178 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:30.572176365 +0000 UTC m=+34.098491082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret") pod "global-pull-secret-syncer-psdng" (UID: "d8692cce-7d57-40d3-9994-7fdfc0adf178") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:14.672957 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:14.672931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:14.673107 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:14.673087    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:14.673182 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:14.673163    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs podName:6d835db3-17d5-457e-a997-a2d7a61b516d nodeName:}" failed. No retries permitted until 2026-09-09 03:06:30.673142517 +0000 UTC m=+34.199457235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs") pod "network-metrics-daemon-6kfmb" (UID: "6d835db3-17d5-457e-a997-a2d7a61b516d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:14.773617 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:14.773511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlgd\" (UniqueName: \"kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd\") pod \"network-check-target-gk5dp\" (UID: \"71a3d2a6-efff-42dc-a3a6-19dbba96041a\") " pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:14.773766 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:14.773667    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:06:14.773766 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:14.773683    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:06:14.773766 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:14.773692    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8tlgd for pod openshift-network-diagnostics/network-check-target-gk5dp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:14.773766 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:14.773743    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd podName:71a3d2a6-efff-42dc-a3a6-19dbba96041a nodeName:}" failed. No retries permitted until 2026-09-09 03:06:30.773730503 +0000 UTC m=+34.300045220 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tlgd" (UniqueName: "kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd") pod "network-check-target-gk5dp" (UID: "71a3d2a6-efff-42dc-a3a6-19dbba96041a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:14.965338 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:14.965301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:14.965499 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:14.965301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:14.965499 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:14.965424    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:14.965629 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:14.965526    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:14.965629 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:14.965310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:14.965721 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:14.965660    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:16.966147 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:16.966116    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:16.966568 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:16.966225    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:16.966568 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:16.966384    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:16.966568 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:16.966476    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:16.966568 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:16.966533    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:16.966773 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:16.966622    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:18.965185 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:18.964942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:18.966046 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:18.965298    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:18.966046 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:18.965010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:18.966046 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:18.964977    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:18.966046 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:18.965381    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:18.966046 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:18.965503    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:19.172038 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.171776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4kpjc" event={"ID":"50cd46eb-3c10-43c8-b520-37f913a814cc","Type":"ContainerStarted","Data":"e6f4d6cf4be070fa876788adfce342d0e815da65cd02f66e444d739f15e7271c"}
Sep 09 03:06:19.174247 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.174223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8" event={"ID":"bb29a572-1fbe-4fba-bc0b-36fb778eaf24","Type":"ContainerStarted","Data":"c2c5af9abaf927f51cf1e8b4f561b39b24af46b64a0e70759dc2e0e751bff11e"}
Sep 09 03:06:19.174363 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.174296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8" event={"ID":"bb29a572-1fbe-4fba-bc0b-36fb778eaf24","Type":"ContainerStarted","Data":"35c2cdd8b95debd802d8068458f1f63991332b06a53c5b0ccdd7fc590fe89847"}
Sep 09 03:06:19.174363 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.174344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8" event={"ID":"bb29a572-1fbe-4fba-bc0b-36fb778eaf24","Type":"ContainerStarted","Data":"9b88fec5972f9e5f551ad6ac204a5f236d08c9d0ac6fa6bbcf15d6a5acdab104"}
Sep 09 03:06:19.174471 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.174389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8" event={"ID":"bb29a572-1fbe-4fba-bc0b-36fb778eaf24","Type":"ContainerStarted","Data":"d8673825e0396b9d72a8ec54bca2a2ed5b7d3fe24fd425ea1bab71a83a612b4a"}
Sep 09 03:06:19.175673 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.175549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jwqmr" event={"ID":"f4039f93-f06b-4c04-9870-ff92b01a66f4","Type":"ContainerStarted","Data":"972748e48b1bd6b5ca9b0ad72eb42c282e48e6eeb75897968787405f90af5f2d"}
Sep 09 03:06:19.176935 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.176915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d" event={"ID":"d3d422be-7e29-4c78-9102-9975a8a3d2ad","Type":"ContainerStarted","Data":"ae5d6857dea56ced1e0336abcf59ceba6d101fddde38fd700234bb34a3bbfe56"}
Sep 09 03:06:19.178157 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.178137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s6hwl" event={"ID":"0d29b378-9477-47b3-9500-c47d907fbe49","Type":"ContainerStarted","Data":"9fec13a0bcaa67e5efcdc9290a864c9960924c306e063a906256a67975c5513b"}
Sep 09 03:06:19.179454 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.179431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal" event={"ID":"9ce43aa27923e3198d26bf26fefbfe86","Type":"ContainerStarted","Data":"1a80efc04f390040154fe6f769548b439a72ae518608cb2a05c3258e548e2664"}
Sep 09 03:06:19.180876 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.180857    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8ab8583-facb-4d47-9197-83089e5468b0" containerID="d2c3e06b28e00e77f1075a936c9d9bbfcecf28922c44366d96e6b0606c81d267" exitCode=0
Sep 09 03:06:19.180952 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.180920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72j6p" event={"ID":"a8ab8583-facb-4d47-9197-83089e5468b0","Type":"ContainerDied","Data":"d2c3e06b28e00e77f1075a936c9d9bbfcecf28922c44366d96e6b0606c81d267"}
Sep 09 03:06:19.182482 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.182462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cd7wf" event={"ID":"ee670977-001f-4a64-8c78-4245ad3befe8","Type":"ContainerStarted","Data":"dab10f546c884e5384dbeba9449a212a71466a5fe2205543a162352423ad680c"}
Sep 09 03:06:19.183985 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.183964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z2zb6" event={"ID":"4896cb96-b860-4f7c-aaa8-2abb525ed851","Type":"ContainerStarted","Data":"3fc0211b4f436a0d96e1d8f6637e082c20d2fc4c9616050f125aaaf984dfe2f7"}
Sep 09 03:06:19.186391 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.186350    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4kpjc" podStartSLOduration=2.802627299 podStartE2EDuration="22.186338749s" totalImagesPullingTime="19.38371145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:19.186051189 +0000 UTC m=+22.712365927" watchObservedRunningTime="2026-09-09 03:06:19.186338749 +0000 UTC m=+22.712653487"
Sep 09 03:06:19.186568 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.186543    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal" podStartSLOduration=21.186535457 podStartE2EDuration="21.186535457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:01.138101714 +0000 UTC m=+4.664416456" watchObservedRunningTime="2026-09-09 03:06:19.186535457 +0000 UTC m=+22.712850194"
Sep 09 03:06:19.198592 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.198530    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jwqmr" podStartSLOduration=2.858878842 podStartE2EDuration="22.198515793s" totalImagesPullingTime="19.339636951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:19.197954536 +0000 UTC m=+22.724269275" watchObservedRunningTime="2026-09-09 03:06:19.198515793 +0000 UTC m=+22.724830532"
Sep 09 03:06:19.215609 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.215547    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal" podStartSLOduration=21.215527616 podStartE2EDuration="21.215527616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:19.215414416 +0000 UTC m=+22.741729155" watchObservedRunningTime="2026-09-09 03:06:19.215527616 +0000 UTC m=+22.741842357"
Sep 09 03:06:19.250476 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.250429    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s6hwl" podStartSLOduration=2.890362388 podStartE2EDuration="22.250413063s" totalImagesPullingTime="19.360050675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:19.250123333 +0000 UTC m=+22.776438072" watchObservedRunningTime="2026-09-09 03:06:19.250413063 +0000 UTC m=+22.776727802"
Sep 09 03:06:19.263772 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.263728    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z2zb6" podStartSLOduration=10.63400714 podStartE2EDuration="22.263712445s" totalImagesPullingTime="11.629705305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:19.263322881 +0000 UTC m=+22.789637620" watchObservedRunningTime="2026-09-09 03:06:19.263712445 +0000 UTC m=+22.790027184"
Sep 09 03:06:19.282726 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:19.282678    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cd7wf" podStartSLOduration=2.848823939 podStartE2EDuration="22.28266263s" totalImagesPullingTime="19.433838691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:19.282319086 +0000 UTC m=+22.808633839" watchObservedRunningTime="2026-09-09 03:06:19.28266263 +0000 UTC m=+22.808977369"
Sep 09 03:06:20.189999 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:20.189971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8" event={"ID":"bb29a572-1fbe-4fba-bc0b-36fb778eaf24","Type":"ContainerStarted","Data":"df313c7ec9ed7254c2db508475571bc81c913572a0aa32926d9349a684275c15"}
Sep 09 03:06:20.190426 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:20.190072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8" event={"ID":"bb29a572-1fbe-4fba-bc0b-36fb778eaf24","Type":"ContainerStarted","Data":"978c6fd7201d9f78386d183fad9976675b17b243a7131a4dda28b1446df14c1f"}
Sep 09 03:06:20.191887 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:20.191858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fvl79" event={"ID":"da582df0-f11f-4ba7-9be7-b233515183ca","Type":"ContainerStarted","Data":"991c976c696a5c0f9f35483c9cc79aa7da2091b97f954f8aa340c3fb60d53be5"}
Sep 09 03:06:20.208990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:20.208967    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 03:06:20.209137 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:20.208959    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fvl79" podStartSLOduration=3.823685423 podStartE2EDuration="23.208939876s" totalImagesPullingTime="19.385254453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:20.20821579 +0000 UTC m=+23.734530529" watchObservedRunningTime="2026-09-09 03:06:20.208939876 +0000 UTC m=+23.735254616"
Sep 09 03:06:20.913744 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:20.913556    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T03:06:20.208984416Z","UUID":"fc8473dc-2b79-4e91-bf4c-9e99a9893c02","Handler":null,"Name":"","Endpoint":""}
Sep 09 03:06:20.915382 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:20.915262    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 03:06:20.915382 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:20.915299    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 03:06:20.965143 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:20.965111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:20.965316 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:20.965122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:20.965316 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:20.965267    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:20.965648 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:20.965631    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:20.965648 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:20.965633    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:20.965787 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:20.965714    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:21.197379 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:21.197287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d" event={"ID":"d3d422be-7e29-4c78-9102-9975a8a3d2ad","Type":"ContainerStarted","Data":"696939a19faadd52cfc66a3575c3c8efba51000d2c66191f0208013284976e82"}
Sep 09 03:06:21.566128 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:21.566101    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:06:21.566792 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:21.566655    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:06:22.203740 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:22.203703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8" event={"ID":"bb29a572-1fbe-4fba-bc0b-36fb778eaf24","Type":"ContainerStarted","Data":"a4a4527568ac80521fbf658da6d389fa141dd9b88a07997487208e4604ad28e8"}
Sep 09 03:06:22.206273 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:22.206232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d" event={"ID":"d3d422be-7e29-4c78-9102-9975a8a3d2ad","Type":"ContainerStarted","Data":"d926cc7478000799e3c5f3cca82d54dbd80ff18c920520b015c36fb78e3f1d01"}
Sep 09 03:06:22.965555 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:22.965520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:22.965753 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:22.965642    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:22.965753 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:22.965520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:22.965753 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:22.965520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:22.965753 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:22.965719    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:22.965948 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:22.965873    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:23.771230 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:23.771093    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:06:23.771670 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:23.771653    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z2zb6"
Sep 09 03:06:23.783874 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:23.783830    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsl7d" podStartSLOduration=4.957731617 podStartE2EDuration="26.783815137s" totalImagesPullingTime="21.82608352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:22.222818931 +0000 UTC m=+25.749133672" watchObservedRunningTime="2026-09-09 03:06:23.783815137 +0000 UTC m=+27.310129873"
Sep 09 03:06:24.214744 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:24.214712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8" event={"ID":"bb29a572-1fbe-4fba-bc0b-36fb778eaf24","Type":"ContainerStarted","Data":"317ff3a5f87d1f65e80d5d2e0aa0944d9bfa230861bed053f27ce6fe82cea468"}
Sep 09 03:06:24.241467 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:24.241420    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8" podStartSLOduration=7.816359893 podStartE2EDuration="27.241404876s" totalImagesPullingTime="19.425044983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:24.240842952 +0000 UTC m=+27.767157691" watchObservedRunningTime="2026-09-09 03:06:24.241404876 +0000 UTC m=+27.767719607"
Sep 09 03:06:24.965464 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:24.965428    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:24.965464 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:24.965455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:24.966346 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:24.965544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:24.966346 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:24.965688    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:24.966346 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:24.965757    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:24.966346 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:24.965802    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:25.219002 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.218903    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8ab8583-facb-4d47-9197-83089e5468b0" containerID="a67f402c65c73cb9916e4a6eba3596bd3a3f34fd2dbbd3ba8dde09b6e30bb83d" exitCode=0
Sep 09 03:06:25.219318 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.219023    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72j6p" event={"ID":"a8ab8583-facb-4d47-9197-83089e5468b0","Type":"ContainerDied","Data":"a67f402c65c73cb9916e4a6eba3596bd3a3f34fd2dbbd3ba8dde09b6e30bb83d"}
Sep 09 03:06:25.219408 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.219386    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:06:25.219489 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.219471    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:06:25.219626 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.219525    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:06:25.236946 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.236919    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:06:25.237127 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.237108    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:06:25.708716 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.708450    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-psdng"]
Sep 09 03:06:25.708935 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.708817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:25.709010 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:25.708933    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:25.711438 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.711157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6kfmb"]
Sep 09 03:06:25.711438 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.711251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:25.711438 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:25.711341    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:25.711893 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.711868    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gk5dp"]
Sep 09 03:06:25.711982 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:25.711958    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:25.712097 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:25.712068    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:27.965173 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:27.965146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:27.965516 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:27.965146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:27.965516 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:27.965273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:27.965516 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:27.965234    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:27.965516 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:27.965361    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:27.965516 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:27.965438    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:28.228079 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:28.227993    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8ab8583-facb-4d47-9197-83089e5468b0" containerID="f059c4cff859f3e846418e6dfbafe6cc65b319cd28bacc9e63640aa93a874125" exitCode=0
Sep 09 03:06:28.228079 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:28.228038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72j6p" event={"ID":"a8ab8583-facb-4d47-9197-83089e5468b0","Type":"ContainerDied","Data":"f059c4cff859f3e846418e6dfbafe6cc65b319cd28bacc9e63640aa93a874125"}
Sep 09 03:06:29.232296 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:29.232221    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8ab8583-facb-4d47-9197-83089e5468b0" containerID="0336699c0fe4cb17896a3febd29810d3c2c47f077e3c1d93dbd1c793fe23311d" exitCode=0
Sep 09 03:06:29.232644 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:29.232293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72j6p" event={"ID":"a8ab8583-facb-4d47-9197-83089e5468b0","Type":"ContainerDied","Data":"0336699c0fe4cb17896a3febd29810d3c2c47f077e3c1d93dbd1c793fe23311d"}
Sep 09 03:06:29.965231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:29.965191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:29.965231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:29.965223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:29.965470 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:29.965241    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:29.965470 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:29.965313    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:29.965611 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:29.965481    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:29.965678 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:29.965617    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:30.584744 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:30.584702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:30.585196 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:30.584861    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:30.585196 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:30.584936    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret podName:d8692cce-7d57-40d3-9994-7fdfc0adf178 nodeName:}" failed. No retries permitted until 2026-09-09 03:07:02.584916675 +0000 UTC m=+66.111231391 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret") pod "global-pull-secret-syncer-psdng" (UID: "d8692cce-7d57-40d3-9994-7fdfc0adf178") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:30.685767 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:30.685731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:30.685948 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:30.685913    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:30.686002 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:30.685981    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs podName:6d835db3-17d5-457e-a997-a2d7a61b516d nodeName:}" failed. No retries permitted until 2026-09-09 03:07:02.685965356 +0000 UTC m=+66.212280077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs") pod "network-metrics-daemon-6kfmb" (UID: "6d835db3-17d5-457e-a997-a2d7a61b516d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:30.786081 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:30.786042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlgd\" (UniqueName: \"kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd\") pod \"network-check-target-gk5dp\" (UID: \"71a3d2a6-efff-42dc-a3a6-19dbba96041a\") " pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:30.786343 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:30.786242    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:06:30.786343 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:30.786270    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:06:30.786343 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:30.786282    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8tlgd for pod openshift-network-diagnostics/network-check-target-gk5dp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:30.786343 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:30.786341    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd podName:71a3d2a6-efff-42dc-a3a6-19dbba96041a nodeName:}" failed. No retries permitted until 2026-09-09 03:07:02.786324304 +0000 UTC m=+66.312639041 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tlgd" (UniqueName: "kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd") pod "network-check-target-gk5dp" (UID: "71a3d2a6-efff-42dc-a3a6-19dbba96041a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:31.965471 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:31.965283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:31.965938 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:31.965339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:31.965938 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:31.965554    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psdng" podUID="d8692cce-7d57-40d3-9994-7fdfc0adf178"
Sep 09 03:06:31.965938 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:31.965686    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:06:31.965938 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:31.965339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:31.965938 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:31.965818    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gk5dp" podUID="71a3d2a6-efff-42dc-a3a6-19dbba96041a"
Sep 09 03:06:32.631093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.631062    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeReady"
Sep 09 03:06:32.631259 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.631197    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 03:06:32.675311 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.675267    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mn566"]
Sep 09 03:06:32.678357 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.678336    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nm9ch"]
Sep 09 03:06:32.678525 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.678477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.680133 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.680070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nm9ch"
Sep 09 03:06:32.680706 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.680686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8s7tp\""
Sep 09 03:06:32.680801 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.680740    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 03:06:32.680895 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.680883    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 03:06:32.682199 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.682181    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 03:06:32.682386 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.682341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 03:06:32.682851 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.682821    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 03:06:32.683033 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.683001    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fjjq4\""
Sep 09 03:06:32.686917 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.686787    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mn566"]
Sep 09 03:06:32.690811 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.689842    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nm9ch"]
Sep 09 03:06:32.801803 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.801767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-tmp-dir\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.801985 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.801818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-config-volume\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.801985 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.801845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.801985 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.801869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b57g\" (UniqueName: \"kubernetes.io/projected/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-kube-api-access-5b57g\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.801985 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.801887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fa0b01d-e400-4fc5-b7b9-12032ba010ca-cert\") pod \"ingress-canary-nm9ch\" (UID: \"4fa0b01d-e400-4fc5-b7b9-12032ba010ca\") " pod="openshift-ingress-canary/ingress-canary-nm9ch"
Sep 09 03:06:32.801985 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.801923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4fj\" (UniqueName: \"kubernetes.io/projected/4fa0b01d-e400-4fc5-b7b9-12032ba010ca-kube-api-access-tg4fj\") pod \"ingress-canary-nm9ch\" (UID: \"4fa0b01d-e400-4fc5-b7b9-12032ba010ca\") " pod="openshift-ingress-canary/ingress-canary-nm9ch"
Sep 09 03:06:32.902452 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.902360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4fj\" (UniqueName: \"kubernetes.io/projected/4fa0b01d-e400-4fc5-b7b9-12032ba010ca-kube-api-access-tg4fj\") pod \"ingress-canary-nm9ch\" (UID: \"4fa0b01d-e400-4fc5-b7b9-12032ba010ca\") " pod="openshift-ingress-canary/ingress-canary-nm9ch"
Sep 09 03:06:32.902452 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.902449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-tmp-dir\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.902689 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.902475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-config-volume\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.902689 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.902492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.902689 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.902509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b57g\" (UniqueName: \"kubernetes.io/projected/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-kube-api-access-5b57g\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.902689 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.902538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fa0b01d-e400-4fc5-b7b9-12032ba010ca-cert\") pod \"ingress-canary-nm9ch\" (UID: \"4fa0b01d-e400-4fc5-b7b9-12032ba010ca\") " pod="openshift-ingress-canary/ingress-canary-nm9ch"
Sep 09 03:06:32.902910 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:32.902835    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:32.902910 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:32.902902    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls podName:3ed10abb-1e74-433a-8d98-c07d0d7d99ff nodeName:}" failed. No retries permitted until 2026-09-09 03:06:33.402880418 +0000 UTC m=+36.929195158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls") pod "dns-default-mn566" (UID: "3ed10abb-1e74-433a-8d98-c07d0d7d99ff") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:32.903032 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.902910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-tmp-dir\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.903193 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.903166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-config-volume\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.907391 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.907370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fa0b01d-e400-4fc5-b7b9-12032ba010ca-cert\") pod \"ingress-canary-nm9ch\" (UID: \"4fa0b01d-e400-4fc5-b7b9-12032ba010ca\") " pod="openshift-ingress-canary/ingress-canary-nm9ch"
Sep 09 03:06:32.911391 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.911369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b57g\" (UniqueName: \"kubernetes.io/projected/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-kube-api-access-5b57g\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:32.911505 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.911483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4fj\" (UniqueName: \"kubernetes.io/projected/4fa0b01d-e400-4fc5-b7b9-12032ba010ca-kube-api-access-tg4fj\") pod \"ingress-canary-nm9ch\" (UID: \"4fa0b01d-e400-4fc5-b7b9-12032ba010ca\") " pod="openshift-ingress-canary/ingress-canary-nm9ch"
Sep 09 03:06:32.998979 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:32.998945    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nm9ch"
Sep 09 03:06:33.406165 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:33.406126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:33.406314 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:33.406278    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:33.406355 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:33.406346    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls podName:3ed10abb-1e74-433a-8d98-c07d0d7d99ff nodeName:}" failed. No retries permitted until 2026-09-09 03:06:34.40633063 +0000 UTC m=+37.932645347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls") pod "dns-default-mn566" (UID: "3ed10abb-1e74-433a-8d98-c07d0d7d99ff") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:33.561754 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:33.561703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nm9ch"]
Sep 09 03:06:33.565188 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:06:33.565164    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa0b01d_e400_4fc5_b7b9_12032ba010ca.slice/crio-a85944b06610e6e813df22771daf7ed6b5c03cc365c829525173c2fcdf566038 WatchSource:0}: Error finding container a85944b06610e6e813df22771daf7ed6b5c03cc365c829525173c2fcdf566038: Status 404 returned error can't find the container with id a85944b06610e6e813df22771daf7ed6b5c03cc365c829525173c2fcdf566038
Sep 09 03:06:33.965004 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:33.964964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:06:33.965004 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:33.965003    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:06:33.965230 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:33.964964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:06:33.967191 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:33.967173    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:06:33.967327 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:33.967201    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:06:33.968051 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:33.968023    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qqmk2\""
Sep 09 03:06:33.968157 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:33.968075    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:06:33.968157 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:33.968128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:06:33.968347 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:33.968333    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bzb6q\""
Sep 09 03:06:34.246358 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:34.246094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nm9ch" event={"ID":"4fa0b01d-e400-4fc5-b7b9-12032ba010ca","Type":"ContainerStarted","Data":"a85944b06610e6e813df22771daf7ed6b5c03cc365c829525173c2fcdf566038"}
Sep 09 03:06:34.249529 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:34.249502    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8ab8583-facb-4d47-9197-83089e5468b0" containerID="31cc6f759cfaa715bf861bc95a8a0a7364c4404e012fac116be7e40cdfa3980f" exitCode=0
Sep 09 03:06:34.249687 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:34.249543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72j6p" event={"ID":"a8ab8583-facb-4d47-9197-83089e5468b0","Type":"ContainerDied","Data":"31cc6f759cfaa715bf861bc95a8a0a7364c4404e012fac116be7e40cdfa3980f"}
Sep 09 03:06:34.415509 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:34.415481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:34.415668 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:34.415647    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:34.415744 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:34.415733    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls podName:3ed10abb-1e74-433a-8d98-c07d0d7d99ff nodeName:}" failed. No retries permitted until 2026-09-09 03:06:36.415709801 +0000 UTC m=+39.942024523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls") pod "dns-default-mn566" (UID: "3ed10abb-1e74-433a-8d98-c07d0d7d99ff") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:35.255837 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:35.255797    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8ab8583-facb-4d47-9197-83089e5468b0" containerID="cbdbbe64aaeff9d11549844e267c0021514c2034c65b0b51e4c4632da31f70e4" exitCode=0
Sep 09 03:06:35.256248 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:35.255860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72j6p" event={"ID":"a8ab8583-facb-4d47-9197-83089e5468b0","Type":"ContainerDied","Data":"cbdbbe64aaeff9d11549844e267c0021514c2034c65b0b51e4c4632da31f70e4"}
Sep 09 03:06:36.259644 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:36.259604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nm9ch" event={"ID":"4fa0b01d-e400-4fc5-b7b9-12032ba010ca","Type":"ContainerStarted","Data":"794f323d85a0edd57fb458e7d38e0f52bd20d87d0228f6928c957ca55cc21284"}
Sep 09 03:06:36.263202 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:36.263177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72j6p" event={"ID":"a8ab8583-facb-4d47-9197-83089e5468b0","Type":"ContainerStarted","Data":"38d25f8cf4c6a7bf166431b22d57f06574f54d77732ad2c4a2907ecfd71dd554"}
Sep 09 03:06:36.277706 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:36.277653    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nm9ch" podStartSLOduration=1.953936157 podStartE2EDuration="4.277636873s" totalImagesPullingTime="2.323700716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:36.277250698 +0000 UTC m=+39.803565435" watchObservedRunningTime="2026-09-09 03:06:36.277636873 +0000 UTC m=+39.803951608"
Sep 09 03:06:36.295252 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:36.295187    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-72j6p" podStartSLOduration=11.540085581 podStartE2EDuration="39.295168038s" totalImagesPullingTime="27.755082457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:36.294829924 +0000 UTC m=+39.821144662" watchObservedRunningTime="2026-09-09 03:06:36.295168038 +0000 UTC m=+39.821482778"
Sep 09 03:06:36.428925 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:36.428885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:36.429096 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:36.429045    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:36.429144 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:36.429127    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls podName:3ed10abb-1e74-433a-8d98-c07d0d7d99ff nodeName:}" failed. No retries permitted until 2026-09-09 03:06:40.4291072 +0000 UTC m=+43.955421935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls") pod "dns-default-mn566" (UID: "3ed10abb-1e74-433a-8d98-c07d0d7d99ff") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:40.455861 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:40.455816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:40.456279 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:40.455972    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:40.456279 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:40.456039    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls podName:3ed10abb-1e74-433a-8d98-c07d0d7d99ff nodeName:}" failed. No retries permitted until 2026-09-09 03:06:48.456023888 +0000 UTC m=+51.982338632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls") pod "dns-default-mn566" (UID: "3ed10abb-1e74-433a-8d98-c07d0d7d99ff") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:48.512493 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:48.512451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:06:48.512925 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:48.512620    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:48.512925 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:06:48.512684    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls podName:3ed10abb-1e74-433a-8d98-c07d0d7d99ff nodeName:}" failed. No retries permitted until 2026-09-09 03:07:04.512668583 +0000 UTC m=+68.038983299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls") pod "dns-default-mn566" (UID: "3ed10abb-1e74-433a-8d98-c07d0d7d99ff") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:57.234697 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:06:57.234669    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z4g8"
Sep 09 03:07:02.606020 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:02.605964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:07:02.608426 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:02.608402    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:07:02.620475 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:02.620446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8692cce-7d57-40d3-9994-7fdfc0adf178-original-pull-secret\") pod \"global-pull-secret-syncer-psdng\" (UID: \"d8692cce-7d57-40d3-9994-7fdfc0adf178\") " pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:07:02.706908 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:02.706869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:07:02.708939 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:02.708918    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:07:02.717679 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:07:02.717654    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 03:07:02.717778 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:07:02.717725    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs podName:6d835db3-17d5-457e-a997-a2d7a61b516d nodeName:}" failed. No retries permitted until 2026-09-09 03:08:06.717703515 +0000 UTC m=+130.244018249 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs") pod "network-metrics-daemon-6kfmb" (UID: "6d835db3-17d5-457e-a997-a2d7a61b516d") : secret "metrics-daemon-secret" not found
Sep 09 03:07:02.779912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:02.779882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psdng"
Sep 09 03:07:02.808113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:02.808075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlgd\" (UniqueName: \"kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd\") pod \"network-check-target-gk5dp\" (UID: \"71a3d2a6-efff-42dc-a3a6-19dbba96041a\") " pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:07:02.810312 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:02.810289    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:07:02.820975 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:02.820954    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:07:02.832958 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:02.832933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tlgd\" (UniqueName: \"kubernetes.io/projected/71a3d2a6-efff-42dc-a3a6-19dbba96041a-kube-api-access-8tlgd\") pod \"network-check-target-gk5dp\" (UID: \"71a3d2a6-efff-42dc-a3a6-19dbba96041a\") " pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:07:03.047593 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:03.047495    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-psdng"]
Sep 09 03:07:03.056155 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:07:03.056130    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8692cce_7d57_40d3_9994_7fdfc0adf178.slice/crio-d4140fd6f208ee275ae8da2cd6fdd3a9b0569c999319ec6e38f09e51f5c5d6ff WatchSource:0}: Error finding container d4140fd6f208ee275ae8da2cd6fdd3a9b0569c999319ec6e38f09e51f5c5d6ff: Status 404 returned error can't find the container with id d4140fd6f208ee275ae8da2cd6fdd3a9b0569c999319ec6e38f09e51f5c5d6ff
Sep 09 03:07:03.087086 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:03.087060    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qqmk2\""
Sep 09 03:07:03.095805 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:03.095783    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:07:03.327764 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:03.327670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-psdng" event={"ID":"d8692cce-7d57-40d3-9994-7fdfc0adf178","Type":"ContainerStarted","Data":"d4140fd6f208ee275ae8da2cd6fdd3a9b0569c999319ec6e38f09e51f5c5d6ff"}
Sep 09 03:07:03.384267 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:03.384237    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gk5dp"]
Sep 09 03:07:03.387371 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:07:03.387340    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a3d2a6_efff_42dc_a3a6_19dbba96041a.slice/crio-a20bb36537e7e828df55afb6a4f748079201ec819f68821466d28f29d394c59f WatchSource:0}: Error finding container a20bb36537e7e828df55afb6a4f748079201ec819f68821466d28f29d394c59f: Status 404 returned error can't find the container with id a20bb36537e7e828df55afb6a4f748079201ec819f68821466d28f29d394c59f
Sep 09 03:07:04.332921 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:04.332874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gk5dp" event={"ID":"71a3d2a6-efff-42dc-a3a6-19dbba96041a","Type":"ContainerStarted","Data":"a20bb36537e7e828df55afb6a4f748079201ec819f68821466d28f29d394c59f"}
Sep 09 03:07:04.519934 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:04.519890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:07:04.520188 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:07:04.520067    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:07:04.520188 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:07:04.520153    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls podName:3ed10abb-1e74-433a-8d98-c07d0d7d99ff nodeName:}" failed. No retries permitted until 2026-09-09 03:07:36.520129492 +0000 UTC m=+100.046444218 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls") pod "dns-default-mn566" (UID: "3ed10abb-1e74-433a-8d98-c07d0d7d99ff") : secret "dns-default-metrics-tls" not found
Sep 09 03:07:08.344721 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:08.344680    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-psdng" event={"ID":"d8692cce-7d57-40d3-9994-7fdfc0adf178","Type":"ContainerStarted","Data":"1de3eb1f99cb78fa69c3371ca966cb1b18e9ada65a591abc76712a482bad0e24"}
Sep 09 03:07:08.346179 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:08.346157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gk5dp" event={"ID":"71a3d2a6-efff-42dc-a3a6-19dbba96041a","Type":"ContainerStarted","Data":"1ff1b75aa2a2f14f67a250c01567ee347c4776d9ba4ad24ddece309baf130557"}
Sep 09 03:07:08.346294 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:08.346279    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:07:08.358394 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:08.358351    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-psdng" podStartSLOduration=66.616418671 podStartE2EDuration="1m11.358336889s" totalImagesPullingTime="4.741918218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:07:08.357733442 +0000 UTC m=+71.884048181" watchObservedRunningTime="2026-09-09 03:07:08.358336889 +0000 UTC m=+71.884651629"
Sep 09 03:07:08.370304 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:08.370239    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gk5dp" podStartSLOduration=66.96288065 podStartE2EDuration="1m11.3702247s" totalImagesPullingTime="4.40734405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:07:08.36971344 +0000 UTC m=+71.896028179" watchObservedRunningTime="2026-09-09 03:07:08.3702247 +0000 UTC m=+71.896539439"
Sep 09 03:07:36.535780 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:36.535737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:07:36.536225 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:07:36.535844    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:07:36.536225 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:07:36.535896    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls podName:3ed10abb-1e74-433a-8d98-c07d0d7d99ff nodeName:}" failed. No retries permitted until 2026-09-09 03:08:40.535883022 +0000 UTC m=+164.062197753 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls") pod "dns-default-mn566" (UID: "3ed10abb-1e74-433a-8d98-c07d0d7d99ff") : secret "dns-default-metrics-tls" not found
Sep 09 03:07:39.351277 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:07:39.351244    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gk5dp"
Sep 09 03:08:06.748544 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:08:06.748503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:08:06.749118 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:08:06.748647    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 03:08:06.749118 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:08:06.748713    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs podName:6d835db3-17d5-457e-a997-a2d7a61b516d nodeName:}" failed. No retries permitted until 2026-09-09 03:10:08.748698736 +0000 UTC m=+252.275013453 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs") pod "network-metrics-daemon-6kfmb" (UID: "6d835db3-17d5-457e-a997-a2d7a61b516d") : secret "metrics-daemon-secret" not found
Sep 09 03:08:35.692982 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:08:35.692923    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mn566" podUID="3ed10abb-1e74-433a-8d98-c07d0d7d99ff"
Sep 09 03:08:36.553997 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:08:36.553967    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mn566"
Sep 09 03:08:36.974825 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:08:36.974784    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6kfmb" podUID="6d835db3-17d5-457e-a997-a2d7a61b516d"
Sep 09 03:08:40.582187 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:08:40.582145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:08:40.582698 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:08:40.582674    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:08:40.582795 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:08:40.582782    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls podName:3ed10abb-1e74-433a-8d98-c07d0d7d99ff nodeName:}" failed. No retries permitted until 2026-09-09 03:10:42.582759753 +0000 UTC m=+286.109074484 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls") pod "dns-default-mn566" (UID: "3ed10abb-1e74-433a-8d98-c07d0d7d99ff") : secret "dns-default-metrics-tls" not found
Sep 09 03:08:49.965331 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:08:49.965294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:09:36.049309 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.049262    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"]
Sep 09 03:09:36.051213 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.051197    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"]
Sep 09 03:09:36.051324 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.051306    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:36.053166 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.053149    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.055696 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.055677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 03:09:36.055813 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.055696    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:09:36.055813 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.055709    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 03:09:36.055813 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.055715    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 03:09:36.055939 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.055849    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 03:09:36.056093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.056079    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 03:09:36.056143 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.056111    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 03:09:36.056344 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.056322    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 03:09:36.056422 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.056379    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 03:09:36.056422 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.056413    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 03:09:36.057115 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.057098    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wlbjr\""
Sep 09 03:09:36.076157 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.076130    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"]
Sep 09 03:09:36.076760 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.076742    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"]
Sep 09 03:09:36.162477 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.162445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.162477 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.162481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-hub\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.162694 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.162499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.162694 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.162542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.162694 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.162612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lmcg8\" (UID: \"18f1b51c-d381-4cb4-ace6-ebbb5084368f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:36.162694 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.162630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9blv\" (UniqueName: \"kubernetes.io/projected/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-kube-api-access-v9blv\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.162694 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.162644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-ca\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.162694 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.162663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfvxt\" (UniqueName: \"kubernetes.io/projected/18f1b51c-d381-4cb4-ace6-ebbb5084368f-kube-api-access-qfvxt\") pod \"cluster-samples-operator-7cdcfcf76c-lmcg8\" (UID: \"18f1b51c-d381-4cb4-ace6-ebbb5084368f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:36.263645 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.263603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-hub\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.263645 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.263644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.263883 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.263679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.263883 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.263710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lmcg8\" (UID: \"18f1b51c-d381-4cb4-ace6-ebbb5084368f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:36.263883 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.263726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9blv\" (UniqueName: \"kubernetes.io/projected/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-kube-api-access-v9blv\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.263883 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.263741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-ca\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.263883 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.263773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfvxt\" (UniqueName: \"kubernetes.io/projected/18f1b51c-d381-4cb4-ace6-ebbb5084368f-kube-api-access-qfvxt\") pod \"cluster-samples-operator-7cdcfcf76c-lmcg8\" (UID: \"18f1b51c-d381-4cb4-ace6-ebbb5084368f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:36.263883 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.263812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.264151 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:36.263893    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 03:09:36.264151 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:36.263967    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls podName:18f1b51c-d381-4cb4-ace6-ebbb5084368f nodeName:}" failed. No retries permitted until 2026-09-09 03:09:36.763947458 +0000 UTC m=+220.290262176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-lmcg8" (UID: "18f1b51c-d381-4cb4-ace6-ebbb5084368f") : secret "samples-operator-tls" not found
Sep 09 03:09:36.264912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.264881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.266364 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.266337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-ca\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.266528 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.266514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.266570 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.266535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.266703 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.266682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-hub\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.278304 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.278271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9blv\" (UniqueName: \"kubernetes.io/projected/7e06cb17-5b47-4a93-9419-ea17a65cdaa2-kube-api-access-v9blv\") pod \"cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p\" (UID: \"7e06cb17-5b47-4a93-9419-ea17a65cdaa2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.278648 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.278631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfvxt\" (UniqueName: \"kubernetes.io/projected/18f1b51c-d381-4cb4-ace6-ebbb5084368f-kube-api-access-qfvxt\") pod \"cluster-samples-operator-7cdcfcf76c-lmcg8\" (UID: \"18f1b51c-d381-4cb4-ace6-ebbb5084368f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:36.371635 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.371604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:09:36.642035 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.641956    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"]
Sep 09 03:09:36.645025 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:09:36.644996    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e06cb17_5b47_4a93_9419_ea17a65cdaa2.slice/crio-37222b8e06b8952ea00688718d59d29416a0e8357d33246bc3d924ccc85326a3 WatchSource:0}: Error finding container 37222b8e06b8952ea00688718d59d29416a0e8357d33246bc3d924ccc85326a3: Status 404 returned error can't find the container with id 37222b8e06b8952ea00688718d59d29416a0e8357d33246bc3d924ccc85326a3
Sep 09 03:09:36.697807 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.697773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" event={"ID":"7e06cb17-5b47-4a93-9419-ea17a65cdaa2","Type":"ContainerStarted","Data":"37222b8e06b8952ea00688718d59d29416a0e8357d33246bc3d924ccc85326a3"}
Sep 09 03:09:36.766500 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:36.766465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lmcg8\" (UID: \"18f1b51c-d381-4cb4-ace6-ebbb5084368f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:36.766682 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:36.766600    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 03:09:36.766682 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:36.766659    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls podName:18f1b51c-d381-4cb4-ace6-ebbb5084368f nodeName:}" failed. No retries permitted until 2026-09-09 03:09:37.766645158 +0000 UTC m=+221.292959876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-lmcg8" (UID: "18f1b51c-d381-4cb4-ace6-ebbb5084368f") : secret "samples-operator-tls" not found
Sep 09 03:09:37.774869 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:37.774826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lmcg8\" (UID: \"18f1b51c-d381-4cb4-ace6-ebbb5084368f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:37.775284 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:37.775031    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 03:09:37.775284 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:37.775123    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls podName:18f1b51c-d381-4cb4-ace6-ebbb5084368f nodeName:}" failed. No retries permitted until 2026-09-09 03:09:39.775099371 +0000 UTC m=+223.301414090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-lmcg8" (UID: "18f1b51c-d381-4cb4-ace6-ebbb5084368f") : secret "samples-operator-tls" not found
Sep 09 03:09:39.790664 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:39.790626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lmcg8\" (UID: \"18f1b51c-d381-4cb4-ace6-ebbb5084368f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:39.791134 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:39.790817    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 03:09:39.791134 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:39.790901    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls podName:18f1b51c-d381-4cb4-ace6-ebbb5084368f nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.790875871 +0000 UTC m=+227.317190612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-lmcg8" (UID: "18f1b51c-d381-4cb4-ace6-ebbb5084368f") : secret "samples-operator-tls" not found
Sep 09 03:09:41.714122 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:41.714085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" event={"ID":"7e06cb17-5b47-4a93-9419-ea17a65cdaa2","Type":"ContainerStarted","Data":"3dd4cf739aaebe86a4e085bd07174503fde7b6cf2d42f9db1addbfb8b0e80cdd"}
Sep 09 03:09:41.714122 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:41.714127    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" event={"ID":"7e06cb17-5b47-4a93-9419-ea17a65cdaa2","Type":"ContainerStarted","Data":"687e598e2bbe7884a3fe37c5e71bb49eb2a7164a546b2aa6107440c21b39e924"}
Sep 09 03:09:41.714643 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:41.714139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" event={"ID":"7e06cb17-5b47-4a93-9419-ea17a65cdaa2","Type":"ContainerStarted","Data":"dac3f5ecafd23cfa9c6036cafa79fcfe25fb613e747e4b08a99327682b3b7066"}
Sep 09 03:09:41.733106 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:41.733064    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" podStartSLOduration=1.606132968 podStartE2EDuration="5.733049245s" totalImagesPullingTime="4.126916277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:41.731860149 +0000 UTC m=+225.258174888" watchObservedRunningTime="2026-09-09 03:09:41.733049245 +0000 UTC m=+225.259363983"
Sep 09 03:09:42.212618 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.212569    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-46d6q"]
Sep 09 03:09:42.214605 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.214568    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.216699 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.216669    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 03:09:42.216871 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.216839    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 03:09:42.216967 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.216887    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 03:09:42.216967 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.216892    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 03:09:42.217832 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.217812    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89spd\""
Sep 09 03:09:42.227200 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.227180    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-46d6q"]
Sep 09 03:09:42.309497 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.309452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.309686 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.309527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5e2ce87-6721-4f8b-9daa-dcbaea830480-crio-socket\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.309686 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.309591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ce87-6721-4f8b-9daa-dcbaea830480-data-volume\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.309686 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.309614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqx9z\" (UniqueName: \"kubernetes.io/projected/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-api-access-sqx9z\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.309686 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.309631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.410038 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.409999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx9z\" (UniqueName: \"kubernetes.io/projected/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-api-access-sqx9z\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.410038 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.410037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.410179 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.410063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.410179 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:42.410141    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:42.410179 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:42.410158    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:42.410297 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.410209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5e2ce87-6721-4f8b-9daa-dcbaea830480-crio-socket\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.410297 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:42.410219    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls podName:d5e2ce87-6721-4f8b-9daa-dcbaea830480 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:42.910200168 +0000 UTC m=+226.436514886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46d6q" (UID: "d5e2ce87-6721-4f8b-9daa-dcbaea830480") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:42.410297 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.410264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ce87-6721-4f8b-9daa-dcbaea830480-data-volume\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.410297 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:42.410276    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm podName:d5e2ce87-6721-4f8b-9daa-dcbaea830480 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:42.910264404 +0000 UTC m=+226.436579121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46d6q" (UID: "d5e2ce87-6721-4f8b-9daa-dcbaea830480") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:42.410466 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.410343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5e2ce87-6721-4f8b-9daa-dcbaea830480-crio-socket\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.410601 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.410564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ce87-6721-4f8b-9daa-dcbaea830480-data-volume\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.418305 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.418285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqx9z\" (UniqueName: \"kubernetes.io/projected/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-api-access-sqx9z\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.914500 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.914462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.914938 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:42.914508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:42.914938 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:42.914607    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:42.914938 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:42.914629    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:42.914938 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:42.914686    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls podName:d5e2ce87-6721-4f8b-9daa-dcbaea830480 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.914668021 +0000 UTC m=+227.440982744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46d6q" (UID: "d5e2ce87-6721-4f8b-9daa-dcbaea830480") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:42.914938 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:42.914706    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm podName:d5e2ce87-6721-4f8b-9daa-dcbaea830480 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.914697699 +0000 UTC m=+227.441012416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46d6q" (UID: "d5e2ce87-6721-4f8b-9daa-dcbaea830480") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:43.821788 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:43.821751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lmcg8\" (UID: \"18f1b51c-d381-4cb4-ace6-ebbb5084368f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:43.821965 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:43.821903    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 03:09:43.822007 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:43.821967    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls podName:18f1b51c-d381-4cb4-ace6-ebbb5084368f nodeName:}" failed. No retries permitted until 2026-09-09 03:09:51.821951945 +0000 UTC m=+235.348266663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-lmcg8" (UID: "18f1b51c-d381-4cb4-ace6-ebbb5084368f") : secret "samples-operator-tls" not found
Sep 09 03:09:43.922882 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:43.922851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:43.923205 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:43.922932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:43.923205 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:43.922988    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:43.923205 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:43.922989    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:43.923205 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:43.923047    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls podName:d5e2ce87-6721-4f8b-9daa-dcbaea830480 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:45.923033298 +0000 UTC m=+229.449348016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46d6q" (UID: "d5e2ce87-6721-4f8b-9daa-dcbaea830480") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:43.923205 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:43.923060    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm podName:d5e2ce87-6721-4f8b-9daa-dcbaea830480 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:45.923054392 +0000 UTC m=+229.449369109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46d6q" (UID: "d5e2ce87-6721-4f8b-9daa-dcbaea830480") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:44.373843 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:44.373814    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s6hwl_0d29b378-9477-47b3-9500-c47d907fbe49/dns-node-resolver/0.log"
Sep 09 03:09:45.174375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:45.174350    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jwqmr_f4039f93-f06b-4c04-9870-ff92b01a66f4/node-ca/0.log"
Sep 09 03:09:45.938502 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:45.938466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:45.938690 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:45.938511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:45.938690 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:45.938611    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:45.938690 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:45.938674    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm podName:d5e2ce87-6721-4f8b-9daa-dcbaea830480 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:49.938658382 +0000 UTC m=+233.464973100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46d6q" (UID: "d5e2ce87-6721-4f8b-9daa-dcbaea830480") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:45.938836 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:45.938617    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:45.938836 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:45.938748    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls podName:d5e2ce87-6721-4f8b-9daa-dcbaea830480 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:49.938735971 +0000 UTC m=+233.465050687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46d6q" (UID: "d5e2ce87-6721-4f8b-9daa-dcbaea830480") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:46.174338 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:46.174308    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nm9ch_4fa0b01d-e400-4fc5-b7b9-12032ba010ca/serve-healthcheck-canary/0.log"
Sep 09 03:09:49.970910 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:49.970878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:49.970910 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:49.970920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:49.971342 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:49.970999    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:49.971342 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:49.971010    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:49.971342 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:49.971063    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls podName:d5e2ce87-6721-4f8b-9daa-dcbaea830480 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:57.971049032 +0000 UTC m=+241.497363749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46d6q" (UID: "d5e2ce87-6721-4f8b-9daa-dcbaea830480") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:49.971342 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:09:49.971075    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm podName:d5e2ce87-6721-4f8b-9daa-dcbaea830480 nodeName:}" failed. No retries permitted until 2026-09-09 03:09:57.971069421 +0000 UTC m=+241.497384138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46d6q" (UID: "d5e2ce87-6721-4f8b-9daa-dcbaea830480") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:51.884327 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:51.884288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lmcg8\" (UID: \"18f1b51c-d381-4cb4-ace6-ebbb5084368f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:51.886774 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:51.886748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f1b51c-d381-4cb4-ace6-ebbb5084368f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lmcg8\" (UID: \"18f1b51c-d381-4cb4-ace6-ebbb5084368f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:51.961874 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:51.961846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"
Sep 09 03:09:52.228035 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:52.227962    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8"]
Sep 09 03:09:52.747916 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:52.747878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8" event={"ID":"18f1b51c-d381-4cb4-ace6-ebbb5084368f","Type":"ContainerStarted","Data":"9a23dc918b0a8940ed1986cdc296479481534c025924f14c95c3371e2894752c"}
Sep 09 03:09:54.755587 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:54.755494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8" event={"ID":"18f1b51c-d381-4cb4-ace6-ebbb5084368f","Type":"ContainerStarted","Data":"9cda69793b1d6e20a3733ee90186c3534a20d37fbf7fc82319f7940dfaf1e8e9"}
Sep 09 03:09:54.755980 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:54.755588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8" event={"ID":"18f1b51c-d381-4cb4-ace6-ebbb5084368f","Type":"ContainerStarted","Data":"defdc7c9925278f7b6e3b1adf68cd61b9961832eb3a1d01a126ea6c0ed55b81d"}
Sep 09 03:09:54.773402 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:54.773347    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lmcg8" podStartSLOduration=16.680008182999998 podStartE2EDuration="18.773328735s" totalImagesPullingTime="2.093320552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:54.772715311 +0000 UTC m=+238.299030050" watchObservedRunningTime="2026-09-09 03:09:54.773328735 +0000 UTC m=+238.299643475"
Sep 09 03:09:58.029234 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:58.029183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:58.029829 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:58.029295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:58.029902 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:58.029879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5e2ce87-6721-4f8b-9daa-dcbaea830480-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:58.031816 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:58.031796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ce87-6721-4f8b-9daa-dcbaea830480-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46d6q\" (UID: \"d5e2ce87-6721-4f8b-9daa-dcbaea830480\") " pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:58.125095 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:58.125062    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89spd\""
Sep 09 03:09:58.133303 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:58.133281    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-46d6q"
Sep 09 03:09:58.429732 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:58.429699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-46d6q"]
Sep 09 03:09:58.433726 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:09:58.433700    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e2ce87_6721_4f8b_9daa_dcbaea830480.slice/crio-61ce20b9e5c16a53ef0484d4e417608ef69131eb9c10846b83de585acc032dbb WatchSource:0}: Error finding container 61ce20b9e5c16a53ef0484d4e417608ef69131eb9c10846b83de585acc032dbb: Status 404 returned error can't find the container with id 61ce20b9e5c16a53ef0484d4e417608ef69131eb9c10846b83de585acc032dbb
Sep 09 03:09:58.773366 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:58.773284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46d6q" event={"ID":"d5e2ce87-6721-4f8b-9daa-dcbaea830480","Type":"ContainerStarted","Data":"1bee390d1db1fd945d5eb6f1184ff2d08b647666dbaf9ab0a581ad4c594db107"}
Sep 09 03:09:58.773366 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:58.773338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46d6q" event={"ID":"d5e2ce87-6721-4f8b-9daa-dcbaea830480","Type":"ContainerStarted","Data":"61ce20b9e5c16a53ef0484d4e417608ef69131eb9c10846b83de585acc032dbb"}
Sep 09 03:09:59.779278 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:09:59.779234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46d6q" event={"ID":"d5e2ce87-6721-4f8b-9daa-dcbaea830480","Type":"ContainerStarted","Data":"17d133b5bef07057e47196427bcbad34533604bfceb6ad0d446a7cbc7f9acd97"}
Sep 09 03:10:00.784258 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:00.784227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46d6q" event={"ID":"d5e2ce87-6721-4f8b-9daa-dcbaea830480","Type":"ContainerStarted","Data":"b75ea637f0920ac99e41e8c39b4016e307c520975ad4cbf6d5ee5b8be83011b4"}
Sep 09 03:10:00.800776 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:00.800680    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-46d6q" podStartSLOduration=16.850757894 podStartE2EDuration="18.800660745s" totalImagesPullingTime="1.949902851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:09:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:00.79997143 +0000 UTC m=+244.326286172" watchObservedRunningTime="2026-09-09 03:10:00.800660745 +0000 UTC m=+244.326975509"
Sep 09 03:10:06.718680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.718647    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"]
Sep 09 03:10:06.720571 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.720555    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:06.723615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.723451    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tcb7j\""
Sep 09 03:10:06.723615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.723467    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 03:10:06.723615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.723485    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 03:10:06.723615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.723500    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 03:10:06.723615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.723514    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 03:10:06.723615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.723463    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 03:10:06.728619 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.728595    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"]
Sep 09 03:10:06.897563 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.897528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/451934bb-acab-4369-9512-200198b619cf-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:06.897774 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.897604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnbjb\" (UniqueName: \"kubernetes.io/projected/451934bb-acab-4369-9512-200198b619cf-kube-api-access-dnbjb\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:06.897774 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.897633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/451934bb-acab-4369-9512-200198b619cf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:06.897774 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.897655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/451934bb-acab-4369-9512-200198b619cf-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:06.998599 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.998469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbjb\" (UniqueName: \"kubernetes.io/projected/451934bb-acab-4369-9512-200198b619cf-kube-api-access-dnbjb\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:06.998599 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.998515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/451934bb-acab-4369-9512-200198b619cf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:06.998848 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.998640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/451934bb-acab-4369-9512-200198b619cf-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:06.998848 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.998721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/451934bb-acab-4369-9512-200198b619cf-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:06.999366 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:06.999287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/451934bb-acab-4369-9512-200198b619cf-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:07.001115 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:07.001085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/451934bb-acab-4369-9512-200198b619cf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:07.001218 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:07.001192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/451934bb-acab-4369-9512-200198b619cf-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:07.006145 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:07.006118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnbjb\" (UniqueName: \"kubernetes.io/projected/451934bb-acab-4369-9512-200198b619cf-kube-api-access-dnbjb\") pod \"prometheus-operator-6f4946db56-cbqdd\" (UID: \"451934bb-acab-4369-9512-200198b619cf\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:07.028958 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:07.028938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"
Sep 09 03:10:07.297918 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:07.297841    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-cbqdd"]
Sep 09 03:10:07.301156 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:10:07.301132    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451934bb_acab_4369_9512_200198b619cf.slice/crio-99788e095915d54a2e71da537e7b486d538071ee059494e1e7cc343c548d24b7 WatchSource:0}: Error finding container 99788e095915d54a2e71da537e7b486d538071ee059494e1e7cc343c548d24b7: Status 404 returned error can't find the container with id 99788e095915d54a2e71da537e7b486d538071ee059494e1e7cc343c548d24b7
Sep 09 03:10:07.807221 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:07.807186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd" event={"ID":"451934bb-acab-4369-9512-200198b619cf","Type":"ContainerStarted","Data":"99788e095915d54a2e71da537e7b486d538071ee059494e1e7cc343c548d24b7"}
Sep 09 03:10:08.811759 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:08.811661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:10:08.812332 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:08.812061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd" event={"ID":"451934bb-acab-4369-9512-200198b619cf","Type":"ContainerStarted","Data":"135d2fc88167b2d6114bb64526da116be49855fcd0af752214bf161398366c69"}
Sep 09 03:10:08.812332 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:08.812105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd" event={"ID":"451934bb-acab-4369-9512-200198b619cf","Type":"ContainerStarted","Data":"83ad405f1a67c99efb2dade911f5675ec1ac3334be21c82e91b656797fe23f3e"}
Sep 09 03:10:08.814200 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:08.814183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d835db3-17d5-457e-a997-a2d7a61b516d-metrics-certs\") pod \"network-metrics-daemon-6kfmb\" (UID: \"6d835db3-17d5-457e-a997-a2d7a61b516d\") " pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:10:08.827740 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:08.827688    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-cbqdd" podStartSLOduration=1.652098353 podStartE2EDuration="2.827669441s" totalImagesPullingTime="1.175571088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:08.826872188 +0000 UTC m=+252.353186940" watchObservedRunningTime="2026-09-09 03:10:08.827669441 +0000 UTC m=+252.353984182"
Sep 09 03:10:08.868167 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:08.868140    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bzb6q\""
Sep 09 03:10:08.876857 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:08.876839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kfmb"
Sep 09 03:10:09.146373 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:09.146340    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6kfmb"]
Sep 09 03:10:09.149406 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:10:09.149376    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d835db3_17d5_457e_a997_a2d7a61b516d.slice/crio-b486e340d597fb143253d7fa11c86fdaeb732b0627140bcdd1f5f63ec9566133 WatchSource:0}: Error finding container b486e340d597fb143253d7fa11c86fdaeb732b0627140bcdd1f5f63ec9566133: Status 404 returned error can't find the container with id b486e340d597fb143253d7fa11c86fdaeb732b0627140bcdd1f5f63ec9566133
Sep 09 03:10:09.817084 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:09.817043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kfmb" event={"ID":"6d835db3-17d5-457e-a997-a2d7a61b516d","Type":"ContainerStarted","Data":"b486e340d597fb143253d7fa11c86fdaeb732b0627140bcdd1f5f63ec9566133"}
Sep 09 03:10:10.823451 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:10.823417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kfmb" event={"ID":"6d835db3-17d5-457e-a997-a2d7a61b516d","Type":"ContainerStarted","Data":"311f0fa12a729fb3a7947660aaa3be498398eb04b98599fb4963f468e74d62fa"}
Sep 09 03:10:10.823835 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:10.823472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kfmb" event={"ID":"6d835db3-17d5-457e-a997-a2d7a61b516d","Type":"ContainerStarted","Data":"4a1e75efb09c81100777e16fa57f23ec5d09581641de8a4c0c323554798832f7"}
Sep 09 03:10:10.846111 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:10.846058    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6kfmb" podStartSLOduration=252.947141792 podStartE2EDuration="4m13.846041387s" totalImagesPullingTime="898.899595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:10.845401859 +0000 UTC m=+254.371716598" watchObservedRunningTime="2026-09-09 03:10:10.846041387 +0000 UTC m=+254.372356125"
Sep 09 03:10:11.103014 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.102982    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"]
Sep 09 03:10:11.106158 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.106140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.108667 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.108637    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 03:10:11.108779 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.108666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z5skz\""
Sep 09 03:10:11.108779 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.108692    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 03:10:11.108779 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.108643    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 03:10:11.109319 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.109300    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bgfkv"]
Sep 09 03:10:11.112340 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.112319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.114543 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.114522    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 03:10:11.114638 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.114602    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 03:10:11.114765 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.114747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 03:10:11.114924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.114904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tzldt\""
Sep 09 03:10:11.119599 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.119559    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"]
Sep 09 03:10:11.227624 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.227624 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxt86\" (UniqueName: \"kubernetes.io/projected/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-api-access-lxt86\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.227888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/482d3f71-6af7-49ac-8483-04240b51ae4d-root\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.227888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-tls\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.227888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/482d3f71-6af7-49ac-8483-04240b51ae4d-sys\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.227888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtxtx\" (UniqueName: \"kubernetes.io/projected/482d3f71-6af7-49ac-8483-04240b51ae4d-kube-api-access-gtxtx\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.227888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-textfile\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.227888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.227888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.227888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.227888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/482d3f71-6af7-49ac-8483-04240b51ae4d-metrics-client-ca\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.228208 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.228208 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.227981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.228208 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.228004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.228208 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.228059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-wtmp\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.329192 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.329372 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.329372 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/482d3f71-6af7-49ac-8483-04240b51ae4d-metrics-client-ca\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.329372 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.329372 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.329372 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.329372 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-wtmp\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.329372 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:10:11.329342    2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 09 03:10:11.329372 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.329785 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxt86\" (UniqueName: \"kubernetes.io/projected/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-api-access-lxt86\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.329785 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:10:11.329417    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-tls podName:1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f nodeName:}" failed. No retries permitted until 2026-09-09 03:10:11.82939653 +0000 UTC m=+255.355711265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-c8nnl" (UID: "1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f") : secret "kube-state-metrics-tls" not found
Sep 09 03:10:11.329785 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/482d3f71-6af7-49ac-8483-04240b51ae4d-root\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.329785 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-tls\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.329785 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/482d3f71-6af7-49ac-8483-04240b51ae4d-sys\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.329785 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtxtx\" (UniqueName: \"kubernetes.io/projected/482d3f71-6af7-49ac-8483-04240b51ae4d-kube-api-access-gtxtx\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.329785 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-textfile\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.329785 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.329785 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.330161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329798    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-wtmp\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.330161 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:10:11.329834    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 03:10:11.330161 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:10:11.329886    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-tls podName:482d3f71-6af7-49ac-8483-04240b51ae4d nodeName:}" failed. No retries permitted until 2026-09-09 03:10:11.829871219 +0000 UTC m=+255.356185951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-tls") pod "node-exporter-bgfkv" (UID: "482d3f71-6af7-49ac-8483-04240b51ae4d") : secret "node-exporter-tls" not found
Sep 09 03:10:11.330161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/482d3f71-6af7-49ac-8483-04240b51ae4d-root\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.330161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/482d3f71-6af7-49ac-8483-04240b51ae4d-metrics-client-ca\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.330161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.329964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.330161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.330026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.330161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.330029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/482d3f71-6af7-49ac-8483-04240b51ae4d-sys\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.330161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.330119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.330441 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.330211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-textfile\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.332170 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.332150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.332381 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.332362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.337122 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.337096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtxtx\" (UniqueName: \"kubernetes.io/projected/482d3f71-6af7-49ac-8483-04240b51ae4d-kube-api-access-gtxtx\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.337779 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.337758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxt86\" (UniqueName: \"kubernetes.io/projected/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-api-access-lxt86\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.834814 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.834782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-tls\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.835224 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.834833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:11.837242 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.837215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/482d3f71-6af7-49ac-8483-04240b51ae4d-node-exporter-tls\") pod \"node-exporter-bgfkv\" (UID: \"482d3f71-6af7-49ac-8483-04240b51ae4d\") " pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:11.837359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:11.837336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-c8nnl\" (UID: \"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:12.016556 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.016520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"
Sep 09 03:10:12.024855 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.023070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bgfkv"
Sep 09 03:10:12.035472 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:10:12.035438    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482d3f71_6af7_49ac_8483_04240b51ae4d.slice/crio-d4e49394cd7ac47171a3a1d60337872914202ed335f66a62daab4eebd3436612 WatchSource:0}: Error finding container d4e49394cd7ac47171a3a1d60337872914202ed335f66a62daab4eebd3436612: Status 404 returned error can't find the container with id d4e49394cd7ac47171a3a1d60337872914202ed335f66a62daab4eebd3436612
Sep 09 03:10:12.132492 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.132462    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:10:12.137092 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.137068    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.139190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.139168    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 03:10:12.139317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.139252    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 03:10:12.139317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.139251    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 03:10:12.139497 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.139477    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 03:10:12.139603 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.139538    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 03:10:12.139603 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.139548    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s6j88\""
Sep 09 03:10:12.139748 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.139659    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 03:10:12.139748 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.139695    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 03:10:12.139957 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.139941    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 03:10:12.140016 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.139941    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 03:10:12.149847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.149823    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:10:12.237876 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.237848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-out\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.237876 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.237880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.238109 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.237895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.238109 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.237915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.238109 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.237943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.238109 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.238033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.238109 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.238071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-web-config\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.238109 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.238097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.238292 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.238147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.238292 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.238176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.238292 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.238234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-volume\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.238292 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.238254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj27m\" (UniqueName: \"kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-kube-api-access-cj27m\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.238292 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.238280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.296724 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.296672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl"]
Sep 09 03:10:12.298680 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:10:12.298655    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf3f4ef_16d7_4ca7_bf47_6991a72b7c6f.slice/crio-7b9f34003cb9e36e8c9da1d97975acdb6a20c97b858d002eff8dbd7ffb6015d3 WatchSource:0}: Error finding container 7b9f34003cb9e36e8c9da1d97975acdb6a20c97b858d002eff8dbd7ffb6015d3: Status 404 returned error can't find the container with id 7b9f34003cb9e36e8c9da1d97975acdb6a20c97b858d002eff8dbd7ffb6015d3
Sep 09 03:10:12.340021 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.339987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-web-config\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-volume\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj27m\" (UniqueName: \"kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-kube-api-access-cj27m\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-out\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.340813 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.340389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.345957 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.344429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.345957 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:10:12.345211    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 03:10:12.345957 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:10:12.345514    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-main-tls podName:8c11a697-ca87-43ad-8e38-61dfe2fe5368 nodeName:}" failed. No retries permitted until 2026-09-09 03:10:12.845482513 +0000 UTC m=+256.371797245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368") : secret "alertmanager-main-tls" not found
Sep 09 03:10:12.345957 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.345476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.345957 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.345678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.347041 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.346551    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-out\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.347264 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.347198    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.347264 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.347203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.347397 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.347321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.349213 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.348319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.349213 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.348844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.349213 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.348952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-volume\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.351934 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.351904    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-web-config\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.352029 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.351982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj27m\" (UniqueName: \"kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-kube-api-access-cj27m\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.832231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.832119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgfkv" event={"ID":"482d3f71-6af7-49ac-8483-04240b51ae4d","Type":"ContainerStarted","Data":"d4e49394cd7ac47171a3a1d60337872914202ed335f66a62daab4eebd3436612"}
Sep 09 03:10:12.833593 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.833554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl" event={"ID":"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f","Type":"ContainerStarted","Data":"7b9f34003cb9e36e8c9da1d97975acdb6a20c97b858d002eff8dbd7ffb6015d3"}
Sep 09 03:10:12.946627 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.946588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:12.949482 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:12.949433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:13.047926 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:13.047883    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:10:13.795691 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:13.795657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:10:13.800119 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:10:13.800085    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c11a697_ca87_43ad_8e38_61dfe2fe5368.slice/crio-b004888117c3501db2c61fa08e445325fda28ce06321c4a1ba5b1cdeaeb2afc3 WatchSource:0}: Error finding container b004888117c3501db2c61fa08e445325fda28ce06321c4a1ba5b1cdeaeb2afc3: Status 404 returned error can't find the container with id b004888117c3501db2c61fa08e445325fda28ce06321c4a1ba5b1cdeaeb2afc3
Sep 09 03:10:13.838189 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:13.838157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerStarted","Data":"b004888117c3501db2c61fa08e445325fda28ce06321c4a1ba5b1cdeaeb2afc3"}
Sep 09 03:10:13.839823 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:13.839797    2580 generic.go:356] "Generic (PLEG): container finished" podID="482d3f71-6af7-49ac-8483-04240b51ae4d" containerID="3d4223c463f2220721d553c2d5d56e6610e543761dfcaa85d232c15f9932b963" exitCode=0
Sep 09 03:10:13.839953 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:13.839878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgfkv" event={"ID":"482d3f71-6af7-49ac-8483-04240b51ae4d","Type":"ContainerDied","Data":"3d4223c463f2220721d553c2d5d56e6610e543761dfcaa85d232c15f9932b963"}
Sep 09 03:10:13.842196 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:13.842167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl" event={"ID":"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f","Type":"ContainerStarted","Data":"cdfe4d862461196c89860c338049e65292bb525d4af815e4d3dbc541040de7bd"}
Sep 09 03:10:13.842298 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:13.842221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl" event={"ID":"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f","Type":"ContainerStarted","Data":"999e145d5d672a017ba033d613e82a4b6d9ebee36c30ac0c01bda9111c649778"}
Sep 09 03:10:13.842298 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:13.842251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl" event={"ID":"1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f","Type":"ContainerStarted","Data":"9ba6e93db76cee6f38bdc177a35e598407b8cd3397f2197c0c9b149c746ec2ac"}
Sep 09 03:10:13.873976 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:13.873894    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-c8nnl" podStartSLOduration=1.652403601 podStartE2EDuration="2.873876284s" totalImagesPullingTime="1.221472683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:13.87320845 +0000 UTC m=+257.399523194" watchObservedRunningTime="2026-09-09 03:10:13.873876284 +0000 UTC m=+257.400191024"
Sep 09 03:10:14.848891 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:14.848816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgfkv" event={"ID":"482d3f71-6af7-49ac-8483-04240b51ae4d","Type":"ContainerStarted","Data":"80ca1b5c4a2def1259acf19b3a70d7c65891211460424b0e911732f9b5928e63"}
Sep 09 03:10:14.848891 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:14.848870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgfkv" event={"ID":"482d3f71-6af7-49ac-8483-04240b51ae4d","Type":"ContainerStarted","Data":"ec347996e75405d63a7e9385d5219aab6df3f74f652c08c43c188d3c50d51524"}
Sep 09 03:10:14.867529 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:14.867482    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bgfkv" podStartSLOduration=2.295470489 podStartE2EDuration="3.86744802s" totalImagesPullingTime="1.48288694s" totalInitContainerRuntime="89.090591ms" isStatefulPod=true podCreationTimestamp="2026-09-09 03:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:14.866086232 +0000 UTC m=+258.392400983" watchObservedRunningTime="2026-09-09 03:10:14.86744802 +0000 UTC m=+258.393762758"
Sep 09 03:10:15.491862 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.491818    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-58bd7d44c7-7fm62"]
Sep 09 03:10:15.494914 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.494894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.497249 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.497227    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 03:10:15.497898 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.497843    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 03:10:15.498096 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.498080    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 03:10:15.498190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.498112    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-58v091kltfg0f\""
Sep 09 03:10:15.498190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.498122    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qkjh7\""
Sep 09 03:10:15.498287 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.498228    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 03:10:15.506688 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.504214    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58bd7d44c7-7fm62"]
Sep 09 03:10:15.564596 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.564542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1029aa7c-77fc-485c-9b25-e048b206a67f-secret-metrics-server-tls\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.564752 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.564605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1029aa7c-77fc-485c-9b25-e048b206a67f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.564752 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.564680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1029aa7c-77fc-485c-9b25-e048b206a67f-secret-metrics-server-client-certs\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.564752 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.564743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1029aa7c-77fc-485c-9b25-e048b206a67f-client-ca-bundle\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.564864 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.564762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvvgc\" (UniqueName: \"kubernetes.io/projected/1029aa7c-77fc-485c-9b25-e048b206a67f-kube-api-access-lvvgc\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.564864 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.564786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1029aa7c-77fc-485c-9b25-e048b206a67f-metrics-server-audit-profiles\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.564864 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.564811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1029aa7c-77fc-485c-9b25-e048b206a67f-audit-log\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.666042 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.666006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1029aa7c-77fc-485c-9b25-e048b206a67f-metrics-server-audit-profiles\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.666042 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.666041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1029aa7c-77fc-485c-9b25-e048b206a67f-audit-log\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.666232 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.666063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1029aa7c-77fc-485c-9b25-e048b206a67f-secret-metrics-server-tls\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.666232 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.666090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1029aa7c-77fc-485c-9b25-e048b206a67f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.666232 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.666131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1029aa7c-77fc-485c-9b25-e048b206a67f-secret-metrics-server-client-certs\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.666232 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.666202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1029aa7c-77fc-485c-9b25-e048b206a67f-client-ca-bundle\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.666232 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.666230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvvgc\" (UniqueName: \"kubernetes.io/projected/1029aa7c-77fc-485c-9b25-e048b206a67f-kube-api-access-lvvgc\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.666537 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.666515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1029aa7c-77fc-485c-9b25-e048b206a67f-audit-log\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.667028 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.666957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1029aa7c-77fc-485c-9b25-e048b206a67f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.667302 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.667110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1029aa7c-77fc-485c-9b25-e048b206a67f-metrics-server-audit-profiles\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.668825 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.668806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1029aa7c-77fc-485c-9b25-e048b206a67f-secret-metrics-server-tls\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.668904 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.668883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1029aa7c-77fc-485c-9b25-e048b206a67f-secret-metrics-server-client-certs\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.668971 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.668927    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1029aa7c-77fc-485c-9b25-e048b206a67f-client-ca-bundle\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.674093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.674072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvvgc\" (UniqueName: \"kubernetes.io/projected/1029aa7c-77fc-485c-9b25-e048b206a67f-kube-api-access-lvvgc\") pod \"metrics-server-58bd7d44c7-7fm62\" (UID: \"1029aa7c-77fc-485c-9b25-e048b206a67f\") " pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.806284 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.806206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:15.854925 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.854887    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerID="1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd" exitCode=0
Sep 09 03:10:15.855257 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:15.854960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerDied","Data":"1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd"}
Sep 09 03:10:16.082523 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:16.082446    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58bd7d44c7-7fm62"]
Sep 09 03:10:16.085862 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:10:16.085829    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1029aa7c_77fc_485c_9b25_e048b206a67f.slice/crio-ec1cff6459941f96f72517e32391b1084094f915dc307a55cd10ce64924f9965 WatchSource:0}: Error finding container ec1cff6459941f96f72517e32391b1084094f915dc307a55cd10ce64924f9965: Status 404 returned error can't find the container with id ec1cff6459941f96f72517e32391b1084094f915dc307a55cd10ce64924f9965
Sep 09 03:10:16.863402 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:16.863361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62" event={"ID":"1029aa7c-77fc-485c-9b25-e048b206a67f","Type":"ContainerStarted","Data":"ec1cff6459941f96f72517e32391b1084094f915dc307a55cd10ce64924f9965"}
Sep 09 03:10:17.292211 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.290467    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:10:17.297353 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.297324    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.300101 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.300070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 03:10:17.300251 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.300075    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 03:10:17.300251 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.300160    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 03:10:17.300251 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.300205    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 03:10:17.300506 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.300491    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 03:10:17.300656 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.300623    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 03:10:17.300770 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.300526    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2r49kaad7kqg7\""
Sep 09 03:10:17.300770 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.300522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 03:10:17.301282 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.301107    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 03:10:17.301282 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.301137    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 03:10:17.301282 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.301279    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 03:10:17.301527 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.301495    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kh592\""
Sep 09 03:10:17.301753 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.301736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 03:10:17.301888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.301786    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 03:10:17.302123 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.302017    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 03:10:17.303190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.303167    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 03:10:17.317108 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.317077    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:10:17.381121 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381121 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jlcx\" (UniqueName: \"kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-kube-api-access-7jlcx\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381594    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.381680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.382122 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.382122 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.382122 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.381734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.484905 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.484870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jlcx\" (UniqueName: \"kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-kube-api-access-7jlcx\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485086 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.484928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485086 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.484964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485086 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485086 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485086 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485289 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485289 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485289 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485289 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485289 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485289 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485289 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485289 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.485289 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.485291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.486086 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.486058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.486298 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.486277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.486364 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.486347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.486423 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.486384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.486474 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.486414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.486474 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.486459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.486600 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.486499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.486600 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.486525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.486986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.486955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.487840 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.487638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.491190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.488232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.491190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.489703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.491190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.489731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.491190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.489970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.491190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.490518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.491190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.490616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.491190 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.491056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.491545 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.491376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.492403 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.492007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.492403 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.492279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.494358 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.494200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.494424 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.494396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.494707 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.494687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.494932 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.494813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.496439 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.496414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jlcx\" (UniqueName: \"kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-kube-api-access-7jlcx\") pod \"prometheus-k8s-0\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:17.610406 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:17.610349    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:18.217568 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:18.217517    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:10:18.220731 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:10:18.220707    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9292ea_5880_42d0_a7bb_1a75d967b8f6.slice/crio-6a117c654d19d0dfd5404cbd4ba639e8201d5c2c0b16bba3bef61fb63f1060bd WatchSource:0}: Error finding container 6a117c654d19d0dfd5404cbd4ba639e8201d5c2c0b16bba3bef61fb63f1060bd: Status 404 returned error can't find the container with id 6a117c654d19d0dfd5404cbd4ba639e8201d5c2c0b16bba3bef61fb63f1060bd
Sep 09 03:10:18.875660 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:18.875620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerStarted","Data":"f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd"}
Sep 09 03:10:18.875824 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:18.875712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerStarted","Data":"c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79"}
Sep 09 03:10:18.875824 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:18.875753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerStarted","Data":"77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa"}
Sep 09 03:10:18.875824 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:18.875789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerStarted","Data":"df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f"}
Sep 09 03:10:18.875955 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:18.875825    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerStarted","Data":"1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51"}
Sep 09 03:10:18.877386 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:18.877353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62" event={"ID":"1029aa7c-77fc-485c-9b25-e048b206a67f","Type":"ContainerStarted","Data":"9c88bfff29bc01d1e12af2082d537be6a5e588ee76edcbaf4cfa73d69399b41e"}
Sep 09 03:10:18.879053 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:18.879026    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerID="2129ca94436eaca8d86a5f391dfe5252dad0ea150f8257bebaeb398072618ebd" exitCode=0
Sep 09 03:10:18.879168 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:18.879084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerDied","Data":"2129ca94436eaca8d86a5f391dfe5252dad0ea150f8257bebaeb398072618ebd"}
Sep 09 03:10:18.879168 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:18.879143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerStarted","Data":"6a117c654d19d0dfd5404cbd4ba639e8201d5c2c0b16bba3bef61fb63f1060bd"}
Sep 09 03:10:18.895402 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:18.895337    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62" podStartSLOduration=2.047178491 podStartE2EDuration="3.895315225s" totalImagesPullingTime="1.848136734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:18.894000174 +0000 UTC m=+262.420314914" watchObservedRunningTime="2026-09-09 03:10:18.895315225 +0000 UTC m=+262.421629965"
Sep 09 03:10:19.886046 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:19.886007    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerStarted","Data":"2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9"}
Sep 09 03:10:19.909210 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:19.909161    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.91160043 podStartE2EDuration="7.90914586s" totalImagesPullingTime="3.98197702s" totalInitContainerRuntime="15.56841ms" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:19.908093439 +0000 UTC m=+263.434408177" watchObservedRunningTime="2026-09-09 03:10:19.90914586 +0000 UTC m=+263.435460590"
Sep 09 03:10:22.904644 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:22.904604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerStarted","Data":"186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea"}
Sep 09 03:10:22.904927 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:22.904685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerStarted","Data":"b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6"}
Sep 09 03:10:24.916978 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:24.916947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerStarted","Data":"cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0"}
Sep 09 03:10:24.917291 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:24.917007    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerStarted","Data":"287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0"}
Sep 09 03:10:24.917291 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:24.917034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerStarted","Data":"447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88"}
Sep 09 03:10:25.924697 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:25.924658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerStarted","Data":"f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367"}
Sep 09 03:10:25.953137 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:25.953084    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.256988077 podStartE2EDuration="8.953057551s" totalImagesPullingTime="5.669705934s" totalInitContainerRuntime="26.36354ms" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:25.951588921 +0000 UTC m=+269.477903652" watchObservedRunningTime="2026-09-09 03:10:25.953057551 +0000 UTC m=+269.479372290"
Sep 09 03:10:27.611465 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:27.611417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:10:35.806350 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:35.806310    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:35.806965 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:35.806381    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:39.555336 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:10:39.555282    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mn566" podUID="3ed10abb-1e74-433a-8d98-c07d0d7d99ff"
Sep 09 03:10:39.978459 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:39.978425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mn566"
Sep 09 03:10:42.600729 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:42.600691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:10:42.603178 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:42.603158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed10abb-1e74-433a-8d98-c07d0d7d99ff-metrics-tls\") pod \"dns-default-mn566\" (UID: \"3ed10abb-1e74-433a-8d98-c07d0d7d99ff\") " pod="openshift-dns/dns-default-mn566"
Sep 09 03:10:42.681693 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:42.681662    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8s7tp\""
Sep 09 03:10:42.690386 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:42.690368    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mn566"
Sep 09 03:10:42.959744 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:42.959705    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mn566"]
Sep 09 03:10:42.962776 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:10:42.962732    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed10abb_1e74_433a_8d98_c07d0d7d99ff.slice/crio-41a908add894608d5d2b10ca0846a59df2ac66c784c9ad1a61449b46eaa65e3b WatchSource:0}: Error finding container 41a908add894608d5d2b10ca0846a59df2ac66c784c9ad1a61449b46eaa65e3b: Status 404 returned error can't find the container with id 41a908add894608d5d2b10ca0846a59df2ac66c784c9ad1a61449b46eaa65e3b
Sep 09 03:10:42.991183 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:42.991143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mn566" event={"ID":"3ed10abb-1e74-433a-8d98-c07d0d7d99ff","Type":"ContainerStarted","Data":"41a908add894608d5d2b10ca0846a59df2ac66c784c9ad1a61449b46eaa65e3b"}
Sep 09 03:10:45.000970 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:45.000933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mn566" event={"ID":"3ed10abb-1e74-433a-8d98-c07d0d7d99ff","Type":"ContainerStarted","Data":"e964b5de3ab318ab8104d7f503e79cbc0d2696998bcba6d776482c2e962bb46c"}
Sep 09 03:10:45.001354 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:45.000982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mn566" event={"ID":"3ed10abb-1e74-433a-8d98-c07d0d7d99ff","Type":"ContainerStarted","Data":"1c6c4bad3c5a049f968ca08c410f9a1ee1094896b8ad9d371ebdf7abdcca4d29"}
Sep 09 03:10:45.001354 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:45.001013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mn566"
Sep 09 03:10:45.015804 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:45.015757    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mn566" podStartSLOduration=251.50827446 podStartE2EDuration="4m13.015740859s" totalImagesPullingTime="1.507466399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:45.015013481 +0000 UTC m=+288.541328232" watchObservedRunningTime="2026-09-09 03:10:45.015740859 +0000 UTC m=+288.542055598"
Sep 09 03:10:55.006895 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:55.006862    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mn566"
Sep 09 03:10:55.812038 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:55.812006    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:55.821475 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:55.821300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-58bd7d44c7-7fm62"
Sep 09 03:10:56.857883 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:10:56.857855    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 03:11:13.710266 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:13.710238    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jwqmr_f4039f93-f06b-4c04-9870-ff92b01a66f4/node-ca/0.log"
Sep 09 03:11:14.710514 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:14.710484    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nm9ch_4fa0b01d-e400-4fc5-b7b9-12032ba010ca/serve-healthcheck-canary/0.log"
Sep 09 03:11:16.372927 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:16.372885    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" podUID="7e06cb17-5b47-4a93-9419-ea17a65cdaa2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 03:11:17.610881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:17.610847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:17.630842 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:17.630808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:18.139166 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:18.139139    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:26.373531 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:26.373481    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" podUID="7e06cb17-5b47-4a93-9419-ea17a65cdaa2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 03:11:31.373284 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:31.373247    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:11:31.373733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:31.373512    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="alertmanager" containerID="cri-o://1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51" gracePeriod=120
Sep 09 03:11:31.373733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:31.373567    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy-metric" containerID="cri-o://f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd" gracePeriod=120
Sep 09 03:11:31.373733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:31.373625    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy-web" containerID="cri-o://77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa" gracePeriod=120
Sep 09 03:11:31.373733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:31.373635    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy" containerID="cri-o://c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79" gracePeriod=120
Sep 09 03:11:31.373733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:31.373655    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="prom-label-proxy" containerID="cri-o://2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9" gracePeriod=120
Sep 09 03:11:31.373733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:31.373669    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="config-reloader" containerID="cri-o://df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f" gracePeriod=120
Sep 09 03:11:32.184491 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.184454    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerID="2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9" exitCode=0
Sep 09 03:11:32.184491 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.184477    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerID="c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79" exitCode=0
Sep 09 03:11:32.184491 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.184484    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerID="df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f" exitCode=0
Sep 09 03:11:32.184491 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.184490    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerID="1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51" exitCode=0
Sep 09 03:11:32.184769 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.184530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerDied","Data":"2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9"}
Sep 09 03:11:32.184769 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.184607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerDied","Data":"c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79"}
Sep 09 03:11:32.184769 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.184635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerDied","Data":"df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f"}
Sep 09 03:11:32.184769 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.184659    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerDied","Data":"1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51"}
Sep 09 03:11:32.616490 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.616465    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:32.737424 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737349    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-volume\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.737424 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737406    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-main-tls\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.737424 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737424    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737454    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737485    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-out\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737508    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-main-db\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737543    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-web\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737599    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-trusted-ca-bundle\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737633    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-web-config\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj27m\" (UniqueName: \"kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-kube-api-access-cj27m\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737702    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-cluster-tls-config\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-metrics-client-ca\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737774    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-tls-assets\") pod \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\" (UID: \"8c11a697-ca87-43ad-8e38-61dfe2fe5368\") "
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.737919    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:11:32.738094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.738079    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-main-db\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.739171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.739104    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:32.739472 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.739414    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:32.740825 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.740689    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:11:32.740825 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.740759    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-out" (OuterVolumeSpecName: "config-out") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:11:32.740825 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.740758    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:32.740825 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.740808    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:32.741329 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.741302    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:32.741520 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.741502    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-kube-api-access-cj27m" (OuterVolumeSpecName: "kube-api-access-cj27m") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "kube-api-access-cj27m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:11:32.741520 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.741500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:32.741790 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.741766    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-volume" (OuterVolumeSpecName: "config-volume") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:32.746545 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.746521    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:32.752636 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.752558    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-web-config" (OuterVolumeSpecName: "web-config") pod "8c11a697-ca87-43ad-8e38-61dfe2fe5368" (UID: "8c11a697-ca87-43ad-8e38-61dfe2fe5368"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:32.838917 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.838887    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-main-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.838917 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.838915    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.839080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.838926    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.839080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.838936    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-out\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.839080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.838944    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.839080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.838952    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.839080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.838978    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-web-config\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.839080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.838987    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj27m\" (UniqueName: \"kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-kube-api-access-cj27m\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.839080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.838995    2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-cluster-tls-config\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.839080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.839003    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c11a697-ca87-43ad-8e38-61dfe2fe5368-metrics-client-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.839080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.839012    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c11a697-ca87-43ad-8e38-61dfe2fe5368-tls-assets\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:32.839080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:32.839020    2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8c11a697-ca87-43ad-8e38-61dfe2fe5368-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:33.191113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.191080    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerID="f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd" exitCode=0
Sep 09 03:11:33.191113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.191106    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerID="77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa" exitCode=0
Sep 09 03:11:33.191345 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.191137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerDied","Data":"f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd"}
Sep 09 03:11:33.191345 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.191161    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.191345 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.191192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerDied","Data":"77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa"}
Sep 09 03:11:33.191345 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.191218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c11a697-ca87-43ad-8e38-61dfe2fe5368","Type":"ContainerDied","Data":"b004888117c3501db2c61fa08e445325fda28ce06321c4a1ba5b1cdeaeb2afc3"}
Sep 09 03:11:33.191345 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.191248    2580 scope.go:117] "RemoveContainer" containerID="2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9"
Sep 09 03:11:33.199541 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.199522    2580 scope.go:117] "RemoveContainer" containerID="f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd"
Sep 09 03:11:33.206390 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.206369    2580 scope.go:117] "RemoveContainer" containerID="c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79"
Sep 09 03:11:33.210569 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.210548    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:11:33.215159 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.215109    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:11:33.216807 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.216777    2580 scope.go:117] "RemoveContainer" containerID="77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa"
Sep 09 03:11:33.224313 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.224297    2580 scope.go:117] "RemoveContainer" containerID="df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f"
Sep 09 03:11:33.232000 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.231980    2580 scope.go:117] "RemoveContainer" containerID="1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51"
Sep 09 03:11:33.236853 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.236829    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:11:33.237183 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237165    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="alertmanager"
Sep 09 03:11:33.237238 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237189    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="alertmanager"
Sep 09 03:11:33.237238 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237207    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="init-config-reloader"
Sep 09 03:11:33.237238 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237216    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="init-config-reloader"
Sep 09 03:11:33.237238 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237234    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="prom-label-proxy"
Sep 09 03:11:33.237355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237242    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="prom-label-proxy"
Sep 09 03:11:33.237355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237254    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="config-reloader"
Sep 09 03:11:33.237355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237261    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="config-reloader"
Sep 09 03:11:33.237355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237268    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy-metric"
Sep 09 03:11:33.237355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237276    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy-metric"
Sep 09 03:11:33.237355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237291    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy-web"
Sep 09 03:11:33.237355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237299    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy-web"
Sep 09 03:11:33.237355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237310    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy"
Sep 09 03:11:33.237355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237318    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy"
Sep 09 03:11:33.237637 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237380    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="alertmanager"
Sep 09 03:11:33.237637 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237389    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy-web"
Sep 09 03:11:33.237637 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237396    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy-metric"
Sep 09 03:11:33.237637 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237408    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="prom-label-proxy"
Sep 09 03:11:33.237637 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237417    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="kube-rbac-proxy"
Sep 09 03:11:33.237637 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.237425    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" containerName="config-reloader"
Sep 09 03:11:33.240904 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.240882    2580 scope.go:117] "RemoveContainer" containerID="1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd"
Sep 09 03:11:33.242861 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.242844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.245172 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.245150    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 03:11:33.245244 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.245185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 03:11:33.245462 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.245448    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 03:11:33.245514 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.245466    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 03:11:33.245514 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.245492    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 03:11:33.245838 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.245814    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 03:11:33.245940 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.245876    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 03:11:33.246055 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.246039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s6j88\""
Sep 09 03:11:33.246278 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.246258    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 03:11:33.249717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.249697    2580 scope.go:117] "RemoveContainer" containerID="2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9"
Sep 09 03:11:33.250060 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:33.250038    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9\": container with ID starting with 2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9 not found: ID does not exist" containerID="2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9"
Sep 09 03:11:33.250152 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.250066    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9"} err="failed to get container status \"2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9\": rpc error: code = NotFound desc = could not find container \"2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9\": container with ID starting with 2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9 not found: ID does not exist"
Sep 09 03:11:33.250152 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.250101    2580 scope.go:117] "RemoveContainer" containerID="f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd"
Sep 09 03:11:33.250434 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:33.250402    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd\": container with ID starting with f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd not found: ID does not exist" containerID="f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd"
Sep 09 03:11:33.250507 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.250446    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd"} err="failed to get container status \"f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd\": rpc error: code = NotFound desc = could not find container \"f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd\": container with ID starting with f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd not found: ID does not exist"
Sep 09 03:11:33.250507 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.250468    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 03:11:33.250507 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.250471    2580 scope.go:117] "RemoveContainer" containerID="c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79"
Sep 09 03:11:33.251096 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:33.250794    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79\": container with ID starting with c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79 not found: ID does not exist" containerID="c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79"
Sep 09 03:11:33.251096 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.250829    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79"} err="failed to get container status \"c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79\": rpc error: code = NotFound desc = could not find container \"c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79\": container with ID starting with c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79 not found: ID does not exist"
Sep 09 03:11:33.251096 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.250843    2580 scope.go:117] "RemoveContainer" containerID="77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa"
Sep 09 03:11:33.251343 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:33.251163    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa\": container with ID starting with 77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa not found: ID does not exist" containerID="77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa"
Sep 09 03:11:33.251343 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.251189    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa"} err="failed to get container status \"77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa\": rpc error: code = NotFound desc = could not find container \"77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa\": container with ID starting with 77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa not found: ID does not exist"
Sep 09 03:11:33.251343 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.251220    2580 scope.go:117] "RemoveContainer" containerID="df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f"
Sep 09 03:11:33.251502 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:33.251481    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f\": container with ID starting with df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f not found: ID does not exist" containerID="df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f"
Sep 09 03:11:33.251559 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.251512    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f"} err="failed to get container status \"df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f\": rpc error: code = NotFound desc = could not find container \"df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f\": container with ID starting with df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f not found: ID does not exist"
Sep 09 03:11:33.251559 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.251533    2580 scope.go:117] "RemoveContainer" containerID="1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51"
Sep 09 03:11:33.251847 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:33.251828    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51\": container with ID starting with 1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51 not found: ID does not exist" containerID="1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51"
Sep 09 03:11:33.251898 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.251850    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51"} err="failed to get container status \"1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51\": rpc error: code = NotFound desc = could not find container \"1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51\": container with ID starting with 1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51 not found: ID does not exist"
Sep 09 03:11:33.251898 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.251865    2580 scope.go:117] "RemoveContainer" containerID="1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd"
Sep 09 03:11:33.252146 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:33.252128    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd\": container with ID starting with 1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd not found: ID does not exist" containerID="1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd"
Sep 09 03:11:33.252208 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.252154    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd"} err="failed to get container status \"1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd\": rpc error: code = NotFound desc = could not find container \"1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd\": container with ID starting with 1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd not found: ID does not exist"
Sep 09 03:11:33.252208 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.252173    2580 scope.go:117] "RemoveContainer" containerID="2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9"
Sep 09 03:11:33.252421 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.252403    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9"} err="failed to get container status \"2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9\": rpc error: code = NotFound desc = could not find container \"2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9\": container with ID starting with 2f5f7bf5f0cab465d1271cb72f3458836b0c8813ff256248563eadad59a352e9 not found: ID does not exist"
Sep 09 03:11:33.252485 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.252422    2580 scope.go:117] "RemoveContainer" containerID="f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd"
Sep 09 03:11:33.252732 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.252715    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd"} err="failed to get container status \"f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd\": rpc error: code = NotFound desc = could not find container \"f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd\": container with ID starting with f5969c3e3437ac46e93c5236c8328392f8031d813044c25fa35af9615d607bcd not found: ID does not exist"
Sep 09 03:11:33.252779 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.252733    2580 scope.go:117] "RemoveContainer" containerID="c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79"
Sep 09 03:11:33.252953 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.252933    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79"} err="failed to get container status \"c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79\": rpc error: code = NotFound desc = could not find container \"c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79\": container with ID starting with c096e473c580e33a9ee0601c6a81b7c7d35be46ca53095cc6ebfd0ae97553d79 not found: ID does not exist"
Sep 09 03:11:33.252999 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.252954    2580 scope.go:117] "RemoveContainer" containerID="77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa"
Sep 09 03:11:33.253194 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.253172    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa"} err="failed to get container status \"77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa\": rpc error: code = NotFound desc = could not find container \"77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa\": container with ID starting with 77852835d54995b7b85aea271741fe6f35f05e9302235bb1b2b457fa880cd9fa not found: ID does not exist"
Sep 09 03:11:33.253263 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.253196    2580 scope.go:117] "RemoveContainer" containerID="df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f"
Sep 09 03:11:33.253459 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.253442    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f"} err="failed to get container status \"df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f\": rpc error: code = NotFound desc = could not find container \"df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f\": container with ID starting with df257fec3a472f9f7fa6914b2219a95bd5793cf179787a3f458ba13d43e9b81f not found: ID does not exist"
Sep 09 03:11:33.253527 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.253460    2580 scope.go:117] "RemoveContainer" containerID="1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51"
Sep 09 03:11:33.253740 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.253719    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51"} err="failed to get container status \"1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51\": rpc error: code = NotFound desc = could not find container \"1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51\": container with ID starting with 1f6198ed5093c07fca37a7af7dd9a06538618f3cdfae2e9280b77766065c5f51 not found: ID does not exist"
Sep 09 03:11:33.253814 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.253741    2580 scope.go:117] "RemoveContainer" containerID="1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd"
Sep 09 03:11:33.253984 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.253963    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd"} err="failed to get container status \"1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd\": rpc error: code = NotFound desc = could not find container \"1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd\": container with ID starting with 1eab4b3d76d112ad7ab493f5e7e6a45c95d1bf6f4835f0fced808641982726fd not found: ID does not exist"
Sep 09 03:11:33.260324 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.260215    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:11:33.341901 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.341868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-web-config\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.341901 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.341903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ae262f6-ff35-426b-a134-f891ad2bacd9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.342131 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.341932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ae262f6-ff35-426b-a134-f891ad2bacd9-config-out\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.342131 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.341973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ae262f6-ff35-426b-a134-f891ad2bacd9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.342131 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.342002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.342131 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.342026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.342131 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.342043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ae262f6-ff35-426b-a134-f891ad2bacd9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.342131 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.342083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.342131 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.342106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.342374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.342144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.342374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.342160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbnt7\" (UniqueName: \"kubernetes.io/projected/3ae262f6-ff35-426b-a134-f891ad2bacd9-kube-api-access-vbnt7\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.342374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.342176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ae262f6-ff35-426b-a134-f891ad2bacd9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.342374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.342201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.442774 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ae262f6-ff35-426b-a134-f891ad2bacd9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.442774 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.442774 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.442774 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ae262f6-ff35-426b-a134-f891ad2bacd9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.443076 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.443076 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.443076 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.443076 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbnt7\" (UniqueName: \"kubernetes.io/projected/3ae262f6-ff35-426b-a134-f891ad2bacd9-kube-api-access-vbnt7\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.443076 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ae262f6-ff35-426b-a134-f891ad2bacd9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.443076 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.443076 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-web-config\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.443076 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ae262f6-ff35-426b-a134-f891ad2bacd9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.443076 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.442991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ae262f6-ff35-426b-a134-f891ad2bacd9-config-out\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.443522 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.443204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ae262f6-ff35-426b-a134-f891ad2bacd9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.444365 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.444027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ae262f6-ff35-426b-a134-f891ad2bacd9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.445280 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.444618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ae262f6-ff35-426b-a134-f891ad2bacd9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.446424 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.446100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ae262f6-ff35-426b-a134-f891ad2bacd9-config-out\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.446424 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.446197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.446424 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.446197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.446424 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.446302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.446424 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.446370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.446424 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.446417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.447139 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.447113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-web-config\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.447361 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.447344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ae262f6-ff35-426b-a134-f891ad2bacd9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.448291 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.448270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ae262f6-ff35-426b-a134-f891ad2bacd9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.452033 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.452016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbnt7\" (UniqueName: \"kubernetes.io/projected/3ae262f6-ff35-426b-a134-f891ad2bacd9-kube-api-access-vbnt7\") pod \"alertmanager-main-0\" (UID: \"3ae262f6-ff35-426b-a134-f891ad2bacd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.554239 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.554206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:11:33.833072 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:33.832945    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:11:33.835813 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:11:33.835784    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae262f6_ff35_426b_a134_f891ad2bacd9.slice/crio-b15802c3bca6b85d2be962c52f8696c20091ae14f3565e246952ad1ac07b4848 WatchSource:0}: Error finding container b15802c3bca6b85d2be962c52f8696c20091ae14f3565e246952ad1ac07b4848: Status 404 returned error can't find the container with id b15802c3bca6b85d2be962c52f8696c20091ae14f3565e246952ad1ac07b4848
Sep 09 03:11:34.201141 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:34.201107    2580 generic.go:356] "Generic (PLEG): container finished" podID="3ae262f6-ff35-426b-a134-f891ad2bacd9" containerID="6910d31f7566a54d86c7ec5677141d1eb8820593fb784f41ac0299b317b16e4b" exitCode=0
Sep 09 03:11:34.201345 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:34.201195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae262f6-ff35-426b-a134-f891ad2bacd9","Type":"ContainerDied","Data":"6910d31f7566a54d86c7ec5677141d1eb8820593fb784f41ac0299b317b16e4b"}
Sep 09 03:11:34.201345 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:34.201278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae262f6-ff35-426b-a134-f891ad2bacd9","Type":"ContainerStarted","Data":"b15802c3bca6b85d2be962c52f8696c20091ae14f3565e246952ad1ac07b4848"}
Sep 09 03:11:34.971313 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:34.971281    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c11a697-ca87-43ad-8e38-61dfe2fe5368" path="/var/lib/kubelet/pods/8c11a697-ca87-43ad-8e38-61dfe2fe5368/volumes"
Sep 09 03:11:35.210034 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.209997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae262f6-ff35-426b-a134-f891ad2bacd9","Type":"ContainerStarted","Data":"e57f213b085152a3c0fc00082ad6989f585b96216305300e81e33ab9d6e005cd"}
Sep 09 03:11:35.210204 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.210057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae262f6-ff35-426b-a134-f891ad2bacd9","Type":"ContainerStarted","Data":"b506af1390739010ec0ef7642797ace667acdb844df4901438c9dfde0a20677f"}
Sep 09 03:11:35.210204 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.210084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae262f6-ff35-426b-a134-f891ad2bacd9","Type":"ContainerStarted","Data":"7c9ae2c5a4495f494f8d3c7d9f204c0dda7799a0807c565c91e86a56a8e09111"}
Sep 09 03:11:35.210204 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.210109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae262f6-ff35-426b-a134-f891ad2bacd9","Type":"ContainerStarted","Data":"c58cf4debecf67e571a0ec28c7dd94c2ff98c80c320e1d7fd8bd313f12456258"}
Sep 09 03:11:35.210204 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.210132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae262f6-ff35-426b-a134-f891ad2bacd9","Type":"ContainerStarted","Data":"7bd6e690f4d2c28302b9bd9311de07342b24e745262a0a23b3f1df3cab02ecc1"}
Sep 09 03:11:35.210204 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.210154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae262f6-ff35-426b-a134-f891ad2bacd9","Type":"ContainerStarted","Data":"8531041a3d3f10fd0ed1cd692866df333b745caadf57f4fd8824eaad992772d2"}
Sep 09 03:11:35.234631 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.234495    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.215585773 podStartE2EDuration="2.234472177s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.886404ms" isStatefulPod=false podCreationTimestamp="2026-09-09 03:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:11:35.233044639 +0000 UTC m=+338.759359393" watchObservedRunningTime="2026-09-09 03:11:35.234472177 +0000 UTC m=+338.760786917"
Sep 09 03:11:35.598448 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.598351    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:11:35.598680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.598641    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="prometheus" containerID="cri-o://b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6" gracePeriod=600
Sep 09 03:11:35.598680 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.598672    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="thanos-sidecar" containerID="cri-o://447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88" gracePeriod=600
Sep 09 03:11:35.598881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.598679    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy" containerID="cri-o://cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0" gracePeriod=600
Sep 09 03:11:35.598881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.598702    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="config-reloader" containerID="cri-o://186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea" gracePeriod=600
Sep 09 03:11:35.598881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.598753    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy-web" containerID="cri-o://287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0" gracePeriod=600
Sep 09 03:11:35.598881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:35.598824    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367" gracePeriod=600
Sep 09 03:11:36.218197 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.218161    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerID="f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367" exitCode=0
Sep 09 03:11:36.218197 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.218186    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerID="cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0" exitCode=0
Sep 09 03:11:36.218197 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.218197    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerID="447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88" exitCode=0
Sep 09 03:11:36.218197 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.218207    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerID="186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea" exitCode=0
Sep 09 03:11:36.218847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.218216    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerID="b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6" exitCode=0
Sep 09 03:11:36.218847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.218230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerDied","Data":"f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367"}
Sep 09 03:11:36.218847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.218290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerDied","Data":"cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0"}
Sep 09 03:11:36.218847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.218316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerDied","Data":"447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88"}
Sep 09 03:11:36.218847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.218340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerDied","Data":"186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea"}
Sep 09 03:11:36.218847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.218363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerDied","Data":"b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6"}
Sep 09 03:11:36.372728 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.372693    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" podUID="7e06cb17-5b47-4a93-9419-ea17a65cdaa2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 03:11:36.372885 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.372760    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p"
Sep 09 03:11:36.373239 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.373217    2580 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"3dd4cf739aaebe86a4e085bd07174503fde7b6cf2d42f9db1addbfb8b0e80cdd"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 03:11:36.373401 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.373263    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" podUID="7e06cb17-5b47-4a93-9419-ea17a65cdaa2" containerName="service-proxy" containerID="cri-o://3dd4cf739aaebe86a4e085bd07174503fde7b6cf2d42f9db1addbfb8b0e80cdd" gracePeriod=30
Sep 09 03:11:36.852299 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.852275    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:36.970909 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.970881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-web-config\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971096 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.970921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config-out\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971096 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.970946    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-db\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971096 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.970976    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-trusted-ca-bundle\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971096 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971007    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-1\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971096 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-serving-certs-ca-bundle\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971096 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-2\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971367 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971100    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-grpc-tls\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971367 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-tls-assets\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971367 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971367 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971193    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jlcx\" (UniqueName: \"kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-kube-api-access-7jlcx\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971367 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971219    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971367 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971256    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-metrics-client-ca\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971367 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971304    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-thanos-prometheus-http-client-file\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971367 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971355    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-tls\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971776 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971362    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:36.971776 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971385    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-kube-rbac-proxy\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971776 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-0\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971776 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971458    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-metrics-client-certs\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971776 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971485    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971776 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-kubelet-serving-ca-bundle\") pod \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\" (UID: \"ff9292ea-5880-42d0-a7bb-1a75d967b8f6\") "
Sep 09 03:11:36.971776 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971609    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:36.972105 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971944    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:36.972105 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.971963    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:36.972200 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.972096    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:36.972419 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.972296    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:36.972518 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.972436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:36.972518 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.972446    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:11:36.974936 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.972725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:36.974936 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.974127    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:36.974936 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.974192    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config-out" (OuterVolumeSpecName: "config-out") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:11:36.975419 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.975387    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-kube-api-access-7jlcx" (OuterVolumeSpecName: "kube-api-access-7jlcx") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "kube-api-access-7jlcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:11:36.975903 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.975859    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:36.976018 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.975910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:36.976254 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.976212    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:36.977194 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.977167    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:36.977443 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.977388    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:36.978056 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.978013    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:36.978056 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.978023    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:11:36.978221 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.978164    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config" (OuterVolumeSpecName: "config") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:36.978301 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.978264    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:36.988487 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:36.988465    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-web-config" (OuterVolumeSpecName: "web-config") pod "ff9292ea-5880-42d0-a7bb-1a75d967b8f6" (UID: "ff9292ea-5880-42d0-a7bb-1a75d967b8f6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:37.072731 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072633    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.072731 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072665    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-web-config\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.072731 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072678    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config-out\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.072731 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072688    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-db\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.072731 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072700    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.072731 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072713    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.072731 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072725    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-grpc-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.072731 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072737    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-tls-assets\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.073107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072748    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.073107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072760    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jlcx\" (UniqueName: \"kubernetes.io/projected/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-kube-api-access-7jlcx\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.073107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072772    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.073107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072800    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-configmap-metrics-client-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.073107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072813    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.073107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072832    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.073107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072846    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-kube-rbac-proxy\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.073107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072859    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.073107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072872    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-secret-metrics-client-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.073107 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.072884    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff9292ea-5880-42d0-a7bb-1a75d967b8f6-config\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:11:37.225433 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.225401    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e06cb17-5b47-4a93-9419-ea17a65cdaa2" containerID="3dd4cf739aaebe86a4e085bd07174503fde7b6cf2d42f9db1addbfb8b0e80cdd" exitCode=2
Sep 09 03:11:37.225827 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.225477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" event={"ID":"7e06cb17-5b47-4a93-9419-ea17a65cdaa2","Type":"ContainerDied","Data":"3dd4cf739aaebe86a4e085bd07174503fde7b6cf2d42f9db1addbfb8b0e80cdd"}
Sep 09 03:11:37.225827 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.225526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65b5f9f7b7-pfk5p" event={"ID":"7e06cb17-5b47-4a93-9419-ea17a65cdaa2","Type":"ContainerStarted","Data":"f54bf439b3368c65f0970404ed4eaccd7f93c1ae6c209c619b9595ada53bbc55"}
Sep 09 03:11:37.228743 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.228717    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerID="287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0" exitCode=0
Sep 09 03:11:37.228877 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.228753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerDied","Data":"287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0"}
Sep 09 03:11:37.228877 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.228819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff9292ea-5880-42d0-a7bb-1a75d967b8f6","Type":"ContainerDied","Data":"6a117c654d19d0dfd5404cbd4ba639e8201d5c2c0b16bba3bef61fb63f1060bd"}
Sep 09 03:11:37.228877 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.228837    2580 scope.go:117] "RemoveContainer" containerID="f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367"
Sep 09 03:11:37.228877 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.228824    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.237047 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.237030    2580 scope.go:117] "RemoveContainer" containerID="cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0"
Sep 09 03:11:37.245661 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.245635    2580 scope.go:117] "RemoveContainer" containerID="287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0"
Sep 09 03:11:37.253237 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.253215    2580 scope.go:117] "RemoveContainer" containerID="447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88"
Sep 09 03:11:37.260417 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.260389    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:11:37.260745 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.260729    2580 scope.go:117] "RemoveContainer" containerID="186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea"
Sep 09 03:11:37.265273 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.265247    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:11:37.268708 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.268690    2580 scope.go:117] "RemoveContainer" containerID="b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6"
Sep 09 03:11:37.276247 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.276232    2580 scope.go:117] "RemoveContainer" containerID="2129ca94436eaca8d86a5f391dfe5252dad0ea150f8257bebaeb398072618ebd"
Sep 09 03:11:37.283414 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.283383    2580 scope.go:117] "RemoveContainer" containerID="f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367"
Sep 09 03:11:37.283704 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:37.283685    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367\": container with ID starting with f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367 not found: ID does not exist" containerID="f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367"
Sep 09 03:11:37.283764 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.283713    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367"} err="failed to get container status \"f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367\": rpc error: code = NotFound desc = could not find container \"f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367\": container with ID starting with f0e7381905563a81325344c767409f703dcd2e9c5032520b8b29d2262622f367 not found: ID does not exist"
Sep 09 03:11:37.283764 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.283733    2580 scope.go:117] "RemoveContainer" containerID="cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0"
Sep 09 03:11:37.283984 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:37.283968    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0\": container with ID starting with cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0 not found: ID does not exist" containerID="cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0"
Sep 09 03:11:37.284021 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.283988    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0"} err="failed to get container status \"cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0\": rpc error: code = NotFound desc = could not find container \"cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0\": container with ID starting with cff4c0a45b3bf551b0143173708c347f13ac8b77c3dec299d4101c9cf966caa0 not found: ID does not exist"
Sep 09 03:11:37.284021 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.284002    2580 scope.go:117] "RemoveContainer" containerID="287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0"
Sep 09 03:11:37.284250 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:37.284232    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0\": container with ID starting with 287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0 not found: ID does not exist" containerID="287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0"
Sep 09 03:11:37.284295 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.284256    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0"} err="failed to get container status \"287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0\": rpc error: code = NotFound desc = could not find container \"287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0\": container with ID starting with 287b5032f515309302e2613c186f7fb9e130679d66aea23f90e46ff1f8c862b0 not found: ID does not exist"
Sep 09 03:11:37.284295 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.284273    2580 scope.go:117] "RemoveContainer" containerID="447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88"
Sep 09 03:11:37.284507 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:37.284491    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88\": container with ID starting with 447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88 not found: ID does not exist" containerID="447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88"
Sep 09 03:11:37.284542 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.284510    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88"} err="failed to get container status \"447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88\": rpc error: code = NotFound desc = could not find container \"447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88\": container with ID starting with 447a2639113bb1b3c586c710114e1bbc4de2247561b01e0f1a425fec94bfbb88 not found: ID does not exist"
Sep 09 03:11:37.284542 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.284523    2580 scope.go:117] "RemoveContainer" containerID="186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea"
Sep 09 03:11:37.284782 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:37.284767    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea\": container with ID starting with 186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea not found: ID does not exist" containerID="186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea"
Sep 09 03:11:37.284846 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.284785    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea"} err="failed to get container status \"186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea\": rpc error: code = NotFound desc = could not find container \"186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea\": container with ID starting with 186b5b3c472443d269f472dd096d7f077aff3821d3b9df5e50800a699449a6ea not found: ID does not exist"
Sep 09 03:11:37.284846 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.284796    2580 scope.go:117] "RemoveContainer" containerID="b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6"
Sep 09 03:11:37.285016 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:37.284999    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6\": container with ID starting with b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6 not found: ID does not exist" containerID="b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6"
Sep 09 03:11:37.285055 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.285021    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6"} err="failed to get container status \"b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6\": rpc error: code = NotFound desc = could not find container \"b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6\": container with ID starting with b3fb86c49eb9c28a50e0a8bd73785fdcbebff4e6bf75dac4a7e88e595e774ac6 not found: ID does not exist"
Sep 09 03:11:37.285055 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.285035    2580 scope.go:117] "RemoveContainer" containerID="2129ca94436eaca8d86a5f391dfe5252dad0ea150f8257bebaeb398072618ebd"
Sep 09 03:11:37.285294 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:11:37.285280    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2129ca94436eaca8d86a5f391dfe5252dad0ea150f8257bebaeb398072618ebd\": container with ID starting with 2129ca94436eaca8d86a5f391dfe5252dad0ea150f8257bebaeb398072618ebd not found: ID does not exist" containerID="2129ca94436eaca8d86a5f391dfe5252dad0ea150f8257bebaeb398072618ebd"
Sep 09 03:11:37.285340 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.285296    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2129ca94436eaca8d86a5f391dfe5252dad0ea150f8257bebaeb398072618ebd"} err="failed to get container status \"2129ca94436eaca8d86a5f391dfe5252dad0ea150f8257bebaeb398072618ebd\": rpc error: code = NotFound desc = could not find container \"2129ca94436eaca8d86a5f391dfe5252dad0ea150f8257bebaeb398072618ebd\": container with ID starting with 2129ca94436eaca8d86a5f391dfe5252dad0ea150f8257bebaeb398072618ebd not found: ID does not exist"
Sep 09 03:11:37.291223 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291205    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:11:37.291480 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291467    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy-thanos"
Sep 09 03:11:37.291532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291481    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy-thanos"
Sep 09 03:11:37.291532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291493    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="init-config-reloader"
Sep 09 03:11:37.291532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291499    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="init-config-reloader"
Sep 09 03:11:37.291532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291505    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="thanos-sidecar"
Sep 09 03:11:37.291532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291510    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="thanos-sidecar"
Sep 09 03:11:37.291532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291517    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="config-reloader"
Sep 09 03:11:37.291532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291522    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="config-reloader"
Sep 09 03:11:37.291532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291528    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="prometheus"
Sep 09 03:11:37.291532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291533    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="prometheus"
Sep 09 03:11:37.291867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291539    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy-web"
Sep 09 03:11:37.291867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291543    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy-web"
Sep 09 03:11:37.291867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291550    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy"
Sep 09 03:11:37.291867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291554    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy"
Sep 09 03:11:37.291867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291615    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy-thanos"
Sep 09 03:11:37.291867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291624    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="config-reloader"
Sep 09 03:11:37.291867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291631    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy"
Sep 09 03:11:37.291867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291638    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="prometheus"
Sep 09 03:11:37.291867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291644    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="thanos-sidecar"
Sep 09 03:11:37.291867 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.291649    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" containerName="kube-rbac-proxy-web"
Sep 09 03:11:37.296785 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.296767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.300152 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.300134    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 03:11:37.300264 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.300224    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kh592\""
Sep 09 03:11:37.300264 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.300232    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 03:11:37.300423 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.300267    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 03:11:37.300593 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.300561    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 03:11:37.300648 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.300602    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 03:11:37.300693 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.300603    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 03:11:37.300943 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.300922    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 03:11:37.301029 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.301011    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 03:11:37.301170 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.301146    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2r49kaad7kqg7\""
Sep 09 03:11:37.301259 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.301245    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 03:11:37.301495 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.301474    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 03:11:37.301754 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.301735    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 03:11:37.302159 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.302139    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 03:11:37.303660 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.303641    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 03:11:37.306071 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.306054    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 03:11:37.311866 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.311847    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:11:37.476273 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476273 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476526 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476526 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476526 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476526 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476526 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476526 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476526 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476526 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476853 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476853 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-config\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476853 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476853 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476853 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476853 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476853 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476853 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476853 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcth\" (UniqueName: \"kubernetes.io/projected/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-kube-api-access-kfcth\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.476853 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.476787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578068 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578068 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-config\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcth\" (UniqueName: \"kubernetes.io/projected/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-kube-api-access-kfcth\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578892 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578892 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578892 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578892 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578892 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578892 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578892 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.578892 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.578526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.579269 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.579183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.579269 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.579212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.579952 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.579889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.581216 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.581173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.581340 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.581241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.581535 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.581511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.581642 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.581593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.581831 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.581809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.582184 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.582156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.582486 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.582455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.582560 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.582457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.582883 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.582856    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.583139 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.583111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.583235 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.583178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-config\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.583235 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.583197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.584263 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.584233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.584358 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.584341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.584424 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.584407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.584710 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.584694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.589466 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.589449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcth\" (UniqueName: \"kubernetes.io/projected/ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d-kube-api-access-kfcth\") pod \"prometheus-k8s-0\" (UID: \"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.606976 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.606953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:11:37.890643 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:37.890615    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:11:37.892146 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:11:37.892117    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5e74ad_e8a7_4064_8b5b_f5eb1f77633d.slice/crio-ecc370f7f721ffce2c98b78fc650e854372df5f29196e46f2ca95b25cd39d0b3 WatchSource:0}: Error finding container ecc370f7f721ffce2c98b78fc650e854372df5f29196e46f2ca95b25cd39d0b3: Status 404 returned error can't find the container with id ecc370f7f721ffce2c98b78fc650e854372df5f29196e46f2ca95b25cd39d0b3
Sep 09 03:11:38.235019 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:38.234935    2580 generic.go:356] "Generic (PLEG): container finished" podID="ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d" containerID="4226457211de176637a0394f92b85284a8de305a2bb9e775a715531237f313d9" exitCode=0
Sep 09 03:11:38.235019 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:38.234983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d","Type":"ContainerDied","Data":"4226457211de176637a0394f92b85284a8de305a2bb9e775a715531237f313d9"}
Sep 09 03:11:38.235397 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:38.235033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d","Type":"ContainerStarted","Data":"ecc370f7f721ffce2c98b78fc650e854372df5f29196e46f2ca95b25cd39d0b3"}
Sep 09 03:11:38.970497 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:38.970460    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff9292ea-5880-42d0-a7bb-1a75d967b8f6" path="/var/lib/kubelet/pods/ff9292ea-5880-42d0-a7bb-1a75d967b8f6/volumes"
Sep 09 03:11:39.243882 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:39.243794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d","Type":"ContainerStarted","Data":"292bae232f7754bde27fc8f7a5af079cdc9cff79be44a441d50428a6c571636c"}
Sep 09 03:11:39.243882 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:39.243868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d","Type":"ContainerStarted","Data":"bb478e3a788c61ef16a45d9561282709de43c2b9e864e6564ac54d1a2b62c8ec"}
Sep 09 03:11:39.244355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:39.243909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d","Type":"ContainerStarted","Data":"93b02affefd51f657e116ef438a9d25f8b89b7eebbeaf31b29a0e7921536ca72"}
Sep 09 03:11:39.244355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:39.243943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d","Type":"ContainerStarted","Data":"594ba97beb102f967fd353a92ab3afcd5bf70f20e98422e3664e990e78e21109"}
Sep 09 03:11:39.244355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:39.243982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d","Type":"ContainerStarted","Data":"cecaa1874b8d4bfef234340d980bbee5ac42f3d5a64cd236ba3215b88bedea64"}
Sep 09 03:11:39.244355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:39.244032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d","Type":"ContainerStarted","Data":"734a8886930cbdf1babf54109a7ff9d4743d826cf62e540711bc0dad08d11284"}
Sep 09 03:11:39.269783 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:39.269725    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.248304872 podStartE2EDuration="2.269665796s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.360924ms" isStatefulPod=false podCreationTimestamp="2026-09-09 03:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:11:39.268613648 +0000 UTC m=+342.794928400" watchObservedRunningTime="2026-09-09 03:11:39.269665796 +0000 UTC m=+342.795980535"
Sep 09 03:11:42.607733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:11:42.607693    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:12:37.608105 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:12:37.608069    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:12:37.624547 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:12:37.624513    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:12:38.487434 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:12:38.487408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:13:35.330363 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.330278    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"]
Sep 09 03:13:35.333793 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.333775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.336839 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.336811    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 03:13:35.337229 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.337212    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 03:13:35.337317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.337270    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 03:13:35.337383 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.337331    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 03:13:35.337440 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.337399    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7r62m\""
Sep 09 03:13:35.347841 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.347816    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"]
Sep 09 03:13:35.360987 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.360958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fcd17b94-6090-4050-ad4e-c6045c1bf678-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.361138 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.361037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcd17b94-6090-4050-ad4e-c6045c1bf678-tmp\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.361138 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.361059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fcd17b94-6090-4050-ad4e-c6045c1bf678-manifests\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.361138 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.361078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp4rl\" (UniqueName: \"kubernetes.io/projected/fcd17b94-6090-4050-ad4e-c6045c1bf678-kube-api-access-xp4rl\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.361138 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.361124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcd17b94-6090-4050-ad4e-c6045c1bf678-webhook-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.461665 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.461627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fcd17b94-6090-4050-ad4e-c6045c1bf678-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.461847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.461730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcd17b94-6090-4050-ad4e-c6045c1bf678-tmp\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.461847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.461763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fcd17b94-6090-4050-ad4e-c6045c1bf678-manifests\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.461847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.461793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp4rl\" (UniqueName: \"kubernetes.io/projected/fcd17b94-6090-4050-ad4e-c6045c1bf678-kube-api-access-xp4rl\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.461847 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.461826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcd17b94-6090-4050-ad4e-c6045c1bf678-webhook-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.462272 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.462240    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fcd17b94-6090-4050-ad4e-c6045c1bf678-manifests\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.464309 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.464274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcd17b94-6090-4050-ad4e-c6045c1bf678-tmp\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.464409 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.464327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcd17b94-6090-4050-ad4e-c6045c1bf678-webhook-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.464409 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.464387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fcd17b94-6090-4050-ad4e-c6045c1bf678-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.477994 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.477964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp4rl\" (UniqueName: \"kubernetes.io/projected/fcd17b94-6090-4050-ad4e-c6045c1bf678-kube-api-access-xp4rl\") pod \"opendatahub-operator-controller-manager-856874fcb4-g9jfv\" (UID: \"fcd17b94-6090-4050-ad4e-c6045c1bf678\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.644099 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.644064    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:35.925783 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.925710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"]
Sep 09 03:13:35.929063 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:13:35.929038    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd17b94_6090_4050_ad4e_c6045c1bf678.slice/crio-d39d67bca42928014cc650c03ea1891e0d663a3152a13e9219926a87d8d59436 WatchSource:0}: Error finding container d39d67bca42928014cc650c03ea1891e0d663a3152a13e9219926a87d8d59436: Status 404 returned error can't find the container with id d39d67bca42928014cc650c03ea1891e0d663a3152a13e9219926a87d8d59436
Sep 09 03:13:35.930948 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:35.930932    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:13:36.700183 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:36.700142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv" event={"ID":"fcd17b94-6090-4050-ad4e-c6045c1bf678","Type":"ContainerStarted","Data":"d39d67bca42928014cc650c03ea1891e0d663a3152a13e9219926a87d8d59436"}
Sep 09 03:13:39.714395 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:39.714361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv" event={"ID":"fcd17b94-6090-4050-ad4e-c6045c1bf678","Type":"ContainerStarted","Data":"1a9d9a1a9e71a0685fcebb1db96bcfd95d464d28b93bfe2fb61155e13d885f51"}
Sep 09 03:13:40.722364 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:40.722332    2580 generic.go:356] "Generic (PLEG): container finished" podID="fcd17b94-6090-4050-ad4e-c6045c1bf678" containerID="1a9d9a1a9e71a0685fcebb1db96bcfd95d464d28b93bfe2fb61155e13d885f51" exitCode=0
Sep 09 03:13:40.722760 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:40.722410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv" event={"ID":"fcd17b94-6090-4050-ad4e-c6045c1bf678","Type":"ContainerDied","Data":"1a9d9a1a9e71a0685fcebb1db96bcfd95d464d28b93bfe2fb61155e13d885f51"}
Sep 09 03:13:41.727980 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:41.727947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv" event={"ID":"fcd17b94-6090-4050-ad4e-c6045c1bf678","Type":"ContainerStarted","Data":"aaa3b6d33eac324ac8d7f7fd861fe2bd01deabbd767a5e1a8383950b1d745974"}
Sep 09 03:13:41.728376 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:41.728114    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:13:41.747361 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:41.747312    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv" podStartSLOduration=2.178467063 podStartE2EDuration="6.747295306s" totalImagesPullingTime="3.492009231s" totalInitContainerRuntime="1.076819012s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:41.746618513 +0000 UTC m=+465.272933251" watchObservedRunningTime="2026-09-09 03:13:41.747295306 +0000 UTC m=+465.273610055"
Sep 09 03:13:52.734112 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:13:52.734081    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-g9jfv"
Sep 09 03:14:12.083590 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.083529    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"]
Sep 09 03:14:12.090131 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.090105    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.092395 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.092364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 03:14:12.092395 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.092365    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 03:14:12.092395 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.092394    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 03:14:12.092690 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.092675    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 03:14:12.092751 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.092725    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 03:14:12.093144 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.093120    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 03:14:12.093219 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.093144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c6p6n\""
Sep 09 03:14:12.099777 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.099758    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"]
Sep 09 03:14:12.172329 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.172295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.172527 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.172341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.172527 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.172385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.172527 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.172416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7ff2e94c-a757-4c85-8473-0c7fe187f545-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.172527 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.172465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff2e94c-a757-4c85-8473-0c7fe187f545-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.172527 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.172498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gpxj\" (UniqueName: \"kubernetes.io/projected/7ff2e94c-a757-4c85-8473-0c7fe187f545-kube-api-access-8gpxj\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.172841 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.172533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.274008 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.273971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.274168 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.274041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.274168 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.274069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.274168 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.274105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.274274 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.274225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7ff2e94c-a757-4c85-8473-0c7fe187f545-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.274315 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.274282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff2e94c-a757-4c85-8473-0c7fe187f545-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.274356 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.274321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gpxj\" (UniqueName: \"kubernetes.io/projected/7ff2e94c-a757-4c85-8473-0c7fe187f545-kube-api-access-8gpxj\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.275260 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.275230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.276561 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.276534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.276720 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.276669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff2e94c-a757-4c85-8473-0c7fe187f545-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.276720 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.276695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.276720 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.276707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7ff2e94c-a757-4c85-8473-0c7fe187f545-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.286790 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.286768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff2e94c-a757-4c85-8473-0c7fe187f545-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.287493 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.287474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gpxj\" (UniqueName: \"kubernetes.io/projected/7ff2e94c-a757-4c85-8473-0c7fe187f545-kube-api-access-8gpxj\") pod \"istiod-openshift-gateway-5677ccb49b-td6vb\" (UID: \"7ff2e94c-a757-4c85-8473-0c7fe187f545\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.400830 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.400785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:12.682401 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.682315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"]
Sep 09 03:14:12.685513 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:14:12.685485    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff2e94c_a757_4c85_8473_0c7fe187f545.slice/crio-f5b22b574b35f98ea9c1cb55a2c4d2d3ed4a5b12a0d7de7dbe23a1ebbbe39fd0 WatchSource:0}: Error finding container f5b22b574b35f98ea9c1cb55a2c4d2d3ed4a5b12a0d7de7dbe23a1ebbbe39fd0: Status 404 returned error can't find the container with id f5b22b574b35f98ea9c1cb55a2c4d2d3ed4a5b12a0d7de7dbe23a1ebbbe39fd0
Sep 09 03:14:12.855463 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:12.855428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb" event={"ID":"7ff2e94c-a757-4c85-8473-0c7fe187f545","Type":"ContainerStarted","Data":"f5b22b574b35f98ea9c1cb55a2c4d2d3ed4a5b12a0d7de7dbe23a1ebbbe39fd0"}
Sep 09 03:14:14.463333 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.463300    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"]
Sep 09 03:14:14.466934 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.466904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.469535 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.469512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 03:14:14.470345 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.470311    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-d7dnd\""
Sep 09 03:14:14.470446 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.470405    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:14:14.470590 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.470449    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 03:14:14.470590 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.470527    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 03:14:14.470717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.470651    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 03:14:14.478920 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.478862    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"]
Sep 09 03:14:14.597593 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.597555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/25c05025-244b-45ff-a9e6-d7123cbde3e4-metrics-cert\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.597799 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.597649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25c05025-244b-45ff-a9e6-d7123cbde3e4-cert\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.597799 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.597756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-875xj\" (UniqueName: \"kubernetes.io/projected/25c05025-244b-45ff-a9e6-d7123cbde3e4-kube-api-access-875xj\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.597799 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.597787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/25c05025-244b-45ff-a9e6-d7123cbde3e4-manager-config\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.698481 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.698445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25c05025-244b-45ff-a9e6-d7123cbde3e4-cert\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.698698 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.698510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-875xj\" (UniqueName: \"kubernetes.io/projected/25c05025-244b-45ff-a9e6-d7123cbde3e4-kube-api-access-875xj\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.698768 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.698712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/25c05025-244b-45ff-a9e6-d7123cbde3e4-manager-config\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.698825 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.698812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/25c05025-244b-45ff-a9e6-d7123cbde3e4-metrics-cert\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.699524 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.699423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/25c05025-244b-45ff-a9e6-d7123cbde3e4-manager-config\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.701956 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.701910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25c05025-244b-45ff-a9e6-d7123cbde3e4-cert\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.702086 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.701987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/25c05025-244b-45ff-a9e6-d7123cbde3e4-metrics-cert\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.714924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.714727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-875xj\" (UniqueName: \"kubernetes.io/projected/25c05025-244b-45ff-a9e6-d7123cbde3e4-kube-api-access-875xj\") pod \"lws-controller-manager-6f6f9bc9b9-9n89j\" (UID: \"25c05025-244b-45ff-a9e6-d7123cbde3e4\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:14.780621 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:14.780568    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:15.345136 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:15.345096    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:14:15.345269 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:15.345168    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 03:14:15.578689 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:15.575348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"]
Sep 09 03:14:15.582618 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:14:15.582570    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c05025_244b_45ff_a9e6_d7123cbde3e4.slice/crio-676b70ef049a701017d37fedfae64a1fdb5669ba241b7f8869419cfa3f057c12 WatchSource:0}: Error finding container 676b70ef049a701017d37fedfae64a1fdb5669ba241b7f8869419cfa3f057c12: Status 404 returned error can't find the container with id 676b70ef049a701017d37fedfae64a1fdb5669ba241b7f8869419cfa3f057c12
Sep 09 03:14:15.869808 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:15.869767    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb" event={"ID":"7ff2e94c-a757-4c85-8473-0c7fe187f545","Type":"ContainerStarted","Data":"2213a8c28b96c0c0c6a7448b226bb4de94c196ff57524a09fa9d5de44f23f36d"}
Sep 09 03:14:15.869992 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:15.869898    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:15.870991 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:15.870968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j" event={"ID":"25c05025-244b-45ff-a9e6-d7123cbde3e4","Type":"ContainerStarted","Data":"676b70ef049a701017d37fedfae64a1fdb5669ba241b7f8869419cfa3f057c12"}
Sep 09 03:14:15.918599 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:15.918510    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb" podStartSLOduration=1.261193816 podStartE2EDuration="3.918486957s" totalImagesPullingTime="2.657293141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:14:15.916671293 +0000 UTC m=+499.442986031" watchObservedRunningTime="2026-09-09 03:14:15.918486957 +0000 UTC m=+499.444801695"
Sep 09 03:14:16.880998 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:16.880968    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-td6vb"
Sep 09 03:14:19.893827 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:19.893792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j" event={"ID":"25c05025-244b-45ff-a9e6-d7123cbde3e4","Type":"ContainerStarted","Data":"531995b47566b241f9e639950560a2b5b8cacb952b69e0d482c4b0c73855f094"}
Sep 09 03:14:19.894264 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:19.893902    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:14:19.914409 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:19.914357    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j" podStartSLOduration=1.820173343 podStartE2EDuration="5.914340327s" totalImagesPullingTime="4.094166984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:14:19.912226625 +0000 UTC m=+503.438541374" watchObservedRunningTime="2026-09-09 03:14:19.914340327 +0000 UTC m=+503.440655128"
Sep 09 03:14:30.900897 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:14:30.900867    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-9n89j"
Sep 09 03:16:16.985435 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:16.985397    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:16:16.992441 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:16.992414    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"
Sep 09 03:16:16.994787 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:16.994763    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 03:16:16.994929 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:16.994763    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 03:16:16.995229 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:16.995212    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 03:16:16.995350 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:16.995256    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4tmns\""
Sep 09 03:16:16.997992 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:16.997971    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:16:17.022872 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:17.022841    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:16:17.067896 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:17.067863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3247983-5d97-40d0-8535-6d666053f16c-config-file\") pod \"limitador-limitador-84994f7ffc-fwzxn\" (UID: \"f3247983-5d97-40d0-8535-6d666053f16c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"
Sep 09 03:16:17.068074 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:17.067911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptpb7\" (UniqueName: \"kubernetes.io/projected/f3247983-5d97-40d0-8535-6d666053f16c-kube-api-access-ptpb7\") pod \"limitador-limitador-84994f7ffc-fwzxn\" (UID: \"f3247983-5d97-40d0-8535-6d666053f16c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"
Sep 09 03:16:17.168461 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:17.168421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3247983-5d97-40d0-8535-6d666053f16c-config-file\") pod \"limitador-limitador-84994f7ffc-fwzxn\" (UID: \"f3247983-5d97-40d0-8535-6d666053f16c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"
Sep 09 03:16:17.168461 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:17.168460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptpb7\" (UniqueName: \"kubernetes.io/projected/f3247983-5d97-40d0-8535-6d666053f16c-kube-api-access-ptpb7\") pod \"limitador-limitador-84994f7ffc-fwzxn\" (UID: \"f3247983-5d97-40d0-8535-6d666053f16c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"
Sep 09 03:16:17.169053 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:17.169033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3247983-5d97-40d0-8535-6d666053f16c-config-file\") pod \"limitador-limitador-84994f7ffc-fwzxn\" (UID: \"f3247983-5d97-40d0-8535-6d666053f16c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"
Sep 09 03:16:17.176683 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:17.176663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptpb7\" (UniqueName: \"kubernetes.io/projected/f3247983-5d97-40d0-8535-6d666053f16c-kube-api-access-ptpb7\") pod \"limitador-limitador-84994f7ffc-fwzxn\" (UID: \"f3247983-5d97-40d0-8535-6d666053f16c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"
Sep 09 03:16:17.304380 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:17.304291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"
Sep 09 03:16:17.579668 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:17.579595    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:16:17.582768 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:16:17.582742    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3247983_5d97_40d0_8535_6d666053f16c.slice/crio-cc97d565f66af8a1d5dc841846ae332abc519782bd43ea62b879d490f45c5fab WatchSource:0}: Error finding container cc97d565f66af8a1d5dc841846ae332abc519782bd43ea62b879d490f45c5fab: Status 404 returned error can't find the container with id cc97d565f66af8a1d5dc841846ae332abc519782bd43ea62b879d490f45c5fab
Sep 09 03:16:18.394371 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:18.394333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn" event={"ID":"f3247983-5d97-40d0-8535-6d666053f16c","Type":"ContainerStarted","Data":"cc97d565f66af8a1d5dc841846ae332abc519782bd43ea62b879d490f45c5fab"}
Sep 09 03:16:20.405261 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:20.405225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn" event={"ID":"f3247983-5d97-40d0-8535-6d666053f16c","Type":"ContainerStarted","Data":"008a17d8ce31243e4524ad7843bf22cb3e67ec885648dd6125b41ab554e41aba"}
Sep 09 03:16:20.405757 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:20.405340    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"
Sep 09 03:16:20.420328 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:20.420280    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn" podStartSLOduration=2.215210999 podStartE2EDuration="4.42025172s" totalImagesPullingTime="2.205040721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:16:20.419384107 +0000 UTC m=+623.945698846" watchObservedRunningTime="2026-09-09 03:16:20.42025172 +0000 UTC m=+623.946566459"
Sep 09 03:16:31.411000 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:16:31.410963    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"
Sep 09 03:17:27.635487 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.635456    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"]
Sep 09 03:17:27.638550 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.638534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"
Sep 09 03:17:27.646474 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.646447    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 03:17:27.646624 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.646520    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ntzrj\""
Sep 09 03:17:27.650962 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.650937    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"]
Sep 09 03:17:27.673827 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.673797    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kmv79"]
Sep 09 03:17:27.676927 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.676908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79"
Sep 09 03:17:27.686060 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.686021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9091a41-7a8b-4c6b-b561-66656c46e9c0-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-69kbd\" (UID: \"a9091a41-7a8b-4c6b-b561-66656c46e9c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"
Sep 09 03:17:27.686351 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.686329    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4j7j\" (UniqueName: \"kubernetes.io/projected/a9091a41-7a8b-4c6b-b561-66656c46e9c0-kube-api-access-b4j7j\") pod \"payload-pre-processing-59fbc47bf8-69kbd\" (UID: \"a9091a41-7a8b-4c6b-b561-66656c46e9c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"
Sep 09 03:17:27.688647 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.688622    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kmv79"]
Sep 09 03:17:27.720125 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.720095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:17:27.787804 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.787771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9091a41-7a8b-4c6b-b561-66656c46e9c0-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-69kbd\" (UID: \"a9091a41-7a8b-4c6b-b561-66656c46e9c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"
Sep 09 03:17:27.787993 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.787831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4j7j\" (UniqueName: \"kubernetes.io/projected/a9091a41-7a8b-4c6b-b561-66656c46e9c0-kube-api-access-b4j7j\") pod \"payload-pre-processing-59fbc47bf8-69kbd\" (UID: \"a9091a41-7a8b-4c6b-b561-66656c46e9c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"
Sep 09 03:17:27.787993 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.787897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96q8b\" (UniqueName: \"kubernetes.io/projected/dcbb770c-adde-492a-92b4-22a3050ced12-kube-api-access-96q8b\") pod \"payload-processing-5b54b7fbb7-kmv79\" (UID: \"dcbb770c-adde-492a-92b4-22a3050ced12\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79"
Sep 09 03:17:27.787993 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.787927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbb770c-adde-492a-92b4-22a3050ced12-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kmv79\" (UID: \"dcbb770c-adde-492a-92b4-22a3050ced12\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79"
Sep 09 03:17:27.788510 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.788489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9091a41-7a8b-4c6b-b561-66656c46e9c0-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-69kbd\" (UID: \"a9091a41-7a8b-4c6b-b561-66656c46e9c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"
Sep 09 03:17:27.795415 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.795393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4j7j\" (UniqueName: \"kubernetes.io/projected/a9091a41-7a8b-4c6b-b561-66656c46e9c0-kube-api-access-b4j7j\") pod \"payload-pre-processing-59fbc47bf8-69kbd\" (UID: \"a9091a41-7a8b-4c6b-b561-66656c46e9c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"
Sep 09 03:17:27.888426 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.888352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96q8b\" (UniqueName: \"kubernetes.io/projected/dcbb770c-adde-492a-92b4-22a3050ced12-kube-api-access-96q8b\") pod \"payload-processing-5b54b7fbb7-kmv79\" (UID: \"dcbb770c-adde-492a-92b4-22a3050ced12\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79"
Sep 09 03:17:27.888426 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.888389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbb770c-adde-492a-92b4-22a3050ced12-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kmv79\" (UID: \"dcbb770c-adde-492a-92b4-22a3050ced12\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79"
Sep 09 03:17:27.889098 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.889079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbb770c-adde-492a-92b4-22a3050ced12-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kmv79\" (UID: \"dcbb770c-adde-492a-92b4-22a3050ced12\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79"
Sep 09 03:17:27.897261 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.897232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96q8b\" (UniqueName: \"kubernetes.io/projected/dcbb770c-adde-492a-92b4-22a3050ced12-kube-api-access-96q8b\") pod \"payload-processing-5b54b7fbb7-kmv79\" (UID: \"dcbb770c-adde-492a-92b4-22a3050ced12\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79"
Sep 09 03:17:27.954971 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.954939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"
Sep 09 03:17:27.988139 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:27.988106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79"
Sep 09 03:17:28.225519 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:28.225490    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"]
Sep 09 03:17:28.227937 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:17:28.227911    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9091a41_7a8b_4c6b_b561_66656c46e9c0.slice/crio-8971025fe30d3db126041b9b598d7c837ee05d196cbc31f2b19baf82430c2f7f WatchSource:0}: Error finding container 8971025fe30d3db126041b9b598d7c837ee05d196cbc31f2b19baf82430c2f7f: Status 404 returned error can't find the container with id 8971025fe30d3db126041b9b598d7c837ee05d196cbc31f2b19baf82430c2f7f
Sep 09 03:17:28.264085 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:28.264057    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kmv79"]
Sep 09 03:17:28.266534 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:17:28.266508    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbb770c_adde_492a_92b4_22a3050ced12.slice/crio-1cc54eefd685dd3f65628e0f4b057521fa7a62008f5a00f0cab7e7ff3f909c6a WatchSource:0}: Error finding container 1cc54eefd685dd3f65628e0f4b057521fa7a62008f5a00f0cab7e7ff3f909c6a: Status 404 returned error can't find the container with id 1cc54eefd685dd3f65628e0f4b057521fa7a62008f5a00f0cab7e7ff3f909c6a
Sep 09 03:17:28.696015 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:28.695975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd" event={"ID":"a9091a41-7a8b-4c6b-b561-66656c46e9c0","Type":"ContainerStarted","Data":"8971025fe30d3db126041b9b598d7c837ee05d196cbc31f2b19baf82430c2f7f"}
Sep 09 03:17:28.697175 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:28.697154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79" event={"ID":"dcbb770c-adde-492a-92b4-22a3050ced12","Type":"ContainerStarted","Data":"1cc54eefd685dd3f65628e0f4b057521fa7a62008f5a00f0cab7e7ff3f909c6a"}
Sep 09 03:17:31.712021 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:31.711988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd" event={"ID":"a9091a41-7a8b-4c6b-b561-66656c46e9c0","Type":"ContainerStarted","Data":"ceac91d1ed907823cd0554ddfc82a8d1d3869af967ecb65c87d679c6be99efaa"}
Sep 09 03:17:31.712474 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:31.712058    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"
Sep 09 03:17:31.713527 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:31.713504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79" event={"ID":"dcbb770c-adde-492a-92b4-22a3050ced12","Type":"ContainerStarted","Data":"bd37befaf3aa1e38ec992be29d6f12a88c31841e848f0a63884823f03b1baa39"}
Sep 09 03:17:31.713657 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:31.713546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79"
Sep 09 03:17:31.728765 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:31.728719    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd" podStartSLOduration=2.273462918 podStartE2EDuration="4.728703467s" totalImagesPullingTime="2.455240549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:31.727640906 +0000 UTC m=+695.253955643" watchObservedRunningTime="2026-09-09 03:17:31.728703467 +0000 UTC m=+695.255018218"
Sep 09 03:17:31.741941 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:31.741892    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79" podStartSLOduration=2.319950094 podStartE2EDuration="4.741866677s" totalImagesPullingTime="2.421916583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:31.740878096 +0000 UTC m=+695.267192845" watchObservedRunningTime="2026-09-09 03:17:31.741866677 +0000 UTC m=+695.268181418"
Sep 09 03:17:42.720022 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:42.719993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-kmv79"
Sep 09 03:17:42.720457 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:42.720059    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-69kbd"
Sep 09 03:17:50.392664 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:50.392628    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:17:55.700963 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:17:55.700930    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:18:05.201870 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:05.201834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:18:21.389074 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:21.389041    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:18:41.494722 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.494692    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"]
Sep 09 03:18:41.497049 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.497031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.499921 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.499895    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 03:18:41.499921 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.499899    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 03:18:41.500084 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.499899    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vxktl\""
Sep 09 03:18:41.500084 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.499948    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 03:18:41.505972 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.505950    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"]
Sep 09 03:18:41.562698 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.562665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e2cb90-ef1e-4741-a44d-693be6c4f83b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.562698 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.562700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.562912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.562736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.562912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.562808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.562912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.562826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwrlq\" (UniqueName: \"kubernetes.io/projected/00e2cb90-ef1e-4741-a44d-693be6c4f83b-kube-api-access-kwrlq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.562912 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.562857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.664176 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.664127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.664176 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.664175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrlq\" (UniqueName: \"kubernetes.io/projected/00e2cb90-ef1e-4741-a44d-693be6c4f83b-kube-api-access-kwrlq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.664445 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.664213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.664445 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.664245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e2cb90-ef1e-4741-a44d-693be6c4f83b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.664445 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.664272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.664445 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.664331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.664733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.664709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.664794 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.664735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.664794 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.664774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.666657 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.666639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e2cb90-ef1e-4741-a44d-693be6c4f83b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.667036 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.667015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e2cb90-ef1e-4741-a44d-693be6c4f83b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.671710 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.671689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwrlq\" (UniqueName: \"kubernetes.io/projected/00e2cb90-ef1e-4741-a44d-693be6c4f83b-kube-api-access-kwrlq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt\" (UID: \"00e2cb90-ef1e-4741-a44d-693be6c4f83b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:41.808259 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:41.808160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:42.079781 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:42.079757    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"]
Sep 09 03:18:42.081884 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:18:42.081860    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e2cb90_ef1e_4741_a44d_693be6c4f83b.slice/crio-4a8334dbab1378912c59e2c72df9c78cd0f30497a678f7fe93958d2efd48c44a WatchSource:0}: Error finding container 4a8334dbab1378912c59e2c72df9c78cd0f30497a678f7fe93958d2efd48c44a: Status 404 returned error can't find the container with id 4a8334dbab1378912c59e2c72df9c78cd0f30497a678f7fe93958d2efd48c44a
Sep 09 03:18:42.083170 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:42.083151    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:18:43.049105 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:43.049077    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt" event={"ID":"00e2cb90-ef1e-4741-a44d-693be6c4f83b","Type":"ContainerStarted","Data":"4a8334dbab1378912c59e2c72df9c78cd0f30497a678f7fe93958d2efd48c44a"}
Sep 09 03:18:43.100175 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:43.100146    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:18:44.055495 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:44.055457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt" event={"ID":"00e2cb90-ef1e-4741-a44d-693be6c4f83b","Type":"ContainerStarted","Data":"521518770ff42356a34cb498879e31f9060c8ba4115bf4c341fd8aadaaf5fc37"}
Sep 09 03:18:44.074196 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:44.074142    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt" podStartSLOduration=2.122843017 podStartE2EDuration="3.074125005s" totalImagesPullingTime="951.281988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:44.072310074 +0000 UTC m=+767.598624835" watchObservedRunningTime="2026-09-09 03:18:44.074125005 +0000 UTC m=+767.600439744"
Sep 09 03:18:50.990961 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:50.990931    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:18:51.809348 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:51.809312    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:51.809556 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:51.809380    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:51.821836 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:51.821811    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:52.102665 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:52.102647    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt"
Sep 09 03:18:57.699840 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:18:57.699804    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:20:20.408306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:20.408267    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:20:21.930269 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:21.930230    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"]
Sep 09 03:20:21.932405 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:21.932389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:20:21.935683 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:21.935664    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b27dde-dockercfg-7vxb9\""
Sep 09 03:20:21.935773 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:21.935662    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b27dde\""
Sep 09 03:20:21.950479 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:21.950449    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"]
Sep 09 03:20:21.981911 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:21.981879    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"]
Sep 09 03:20:21.984029 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:21.984013    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:20:21.995473 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:21.995447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"]
Sep 09 03:20:22.075365 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.075338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltxzb\" (UniqueName: \"kubernetes.io/projected/1c2e3179-a7aa-4f52-a37d-74985be48cf0-kube-api-access-ltxzb\") pod \"payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429\" (UID: \"1c2e3179-a7aa-4f52-a37d-74985be48cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:20:22.075522 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.075407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c2e3179-a7aa-4f52-a37d-74985be48cf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429\" (UID: \"1c2e3179-a7aa-4f52-a37d-74985be48cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:20:22.075522 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.075477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbmg\" (UniqueName: \"kubernetes.io/projected/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-kube-api-access-dwbmg\") pod \"payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl\" (UID: \"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:20:22.075522 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.075519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl\" (UID: \"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:20:22.176444 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.176413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbmg\" (UniqueName: \"kubernetes.io/projected/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-kube-api-access-dwbmg\") pod \"payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl\" (UID: \"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:20:22.176661 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.176455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl\" (UID: \"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:20:22.176661 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.176482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltxzb\" (UniqueName: \"kubernetes.io/projected/1c2e3179-a7aa-4f52-a37d-74985be48cf0-kube-api-access-ltxzb\") pod \"payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429\" (UID: \"1c2e3179-a7aa-4f52-a37d-74985be48cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:20:22.176661 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.176534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c2e3179-a7aa-4f52-a37d-74985be48cf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429\" (UID: \"1c2e3179-a7aa-4f52-a37d-74985be48cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:20:22.177231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.177209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl\" (UID: \"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:20:22.177280 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.177231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c2e3179-a7aa-4f52-a37d-74985be48cf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429\" (UID: \"1c2e3179-a7aa-4f52-a37d-74985be48cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:20:22.184661 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.184600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbmg\" (UniqueName: \"kubernetes.io/projected/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-kube-api-access-dwbmg\") pod \"payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl\" (UID: \"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:20:22.184661 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.184612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltxzb\" (UniqueName: \"kubernetes.io/projected/1c2e3179-a7aa-4f52-a37d-74985be48cf0-kube-api-access-ltxzb\") pod \"payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429\" (UID: \"1c2e3179-a7aa-4f52-a37d-74985be48cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:20:22.242633 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.242598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:20:22.294080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.293885    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:20:22.543035 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.542905    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"]
Sep 09 03:20:22.545881 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:20:22.545847    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c0bebc_ab07_4bc9_a0a1_e3d35dd068d2.slice/crio-85068d4627362c720a88b8e79591cc6949d876ea8d8971f3e691c309282ab1f0 WatchSource:0}: Error finding container 85068d4627362c720a88b8e79591cc6949d876ea8d8971f3e691c309282ab1f0: Status 404 returned error can't find the container with id 85068d4627362c720a88b8e79591cc6949d876ea8d8971f3e691c309282ab1f0
Sep 09 03:20:22.619355 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:22.619006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"]
Sep 09 03:20:22.622265 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:20:22.622227    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2e3179_a7aa_4f52_a37d_74985be48cf0.slice/crio-453d9156d003e4408daf0edcfac4a885c689c0b055e430c21d2e49a39e0d45ec WatchSource:0}: Error finding container 453d9156d003e4408daf0edcfac4a885c689c0b055e430c21d2e49a39e0d45ec: Status 404 returned error can't find the container with id 453d9156d003e4408daf0edcfac4a885c689c0b055e430c21d2e49a39e0d45ec
Sep 09 03:20:23.497131 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:23.497097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl" event={"ID":"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2","Type":"ContainerStarted","Data":"40daf5e06a08f68189bf254e53c9b7ec44ba502f864ec012d27b9b0d656f5674"}
Sep 09 03:20:23.497570 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:23.497156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl" event={"ID":"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2","Type":"ContainerStarted","Data":"85068d4627362c720a88b8e79591cc6949d876ea8d8971f3e691c309282ab1f0"}
Sep 09 03:20:23.497570 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:23.497209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:20:23.498758 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:23.498737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429" event={"ID":"1c2e3179-a7aa-4f52-a37d-74985be48cf0","Type":"ContainerStarted","Data":"952bb87458e67bff8312b58b1f4d2406565836d8b4cd2855c4b6be23f7fb7fe9"}
Sep 09 03:20:23.498899 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:23.498770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429" event={"ID":"1c2e3179-a7aa-4f52-a37d-74985be48cf0","Type":"ContainerStarted","Data":"453d9156d003e4408daf0edcfac4a885c689c0b055e430c21d2e49a39e0d45ec"}
Sep 09 03:20:23.498899 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:23.498860    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:20:23.529332 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:23.529288    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl" podStartSLOduration=2.529276003 podStartE2EDuration="2.529276003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:23.52776244 +0000 UTC m=+867.054077178" watchObservedRunningTime="2026-09-09 03:20:23.529276003 +0000 UTC m=+867.055590775"
Sep 09 03:20:23.545233 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:23.545143    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429" podStartSLOduration=2.54512642 podStartE2EDuration="2.54512642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:23.545027965 +0000 UTC m=+867.071342705" watchObservedRunningTime="2026-09-09 03:20:23.54512642 +0000 UTC m=+867.071441158"
Sep 09 03:20:29.810325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.810291    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"]
Sep 09 03:20:29.812820 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.812804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:20:29.815874 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.815852    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5ecf63f7-dockercfg-pchwz\""
Sep 09 03:20:29.815874 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.815854    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5ecf63f7\""
Sep 09 03:20:29.828151 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.828130    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"]
Sep 09 03:20:29.856650 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.856615    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"]
Sep 09 03:20:29.858879 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.858849    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:20:29.868835 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.868811    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"]
Sep 09 03:20:29.951180 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.951145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfwrq\" (UniqueName: \"kubernetes.io/projected/a3b7cf90-2215-402f-acbf-710f216ac467-kube-api-access-zfwrq\") pod \"payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776\" (UID: \"a3b7cf90-2215-402f-acbf-710f216ac467\") " pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:20:29.951180 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.951181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx\" (UID: \"87d9bde0-b1b2-45f5-a154-ea0a584e66a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:20:29.951386 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.951222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b7cf90-2215-402f-acbf-710f216ac467-plugins-config-volume\") pod \"payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776\" (UID: \"a3b7cf90-2215-402f-acbf-710f216ac467\") " pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:20:29.951386 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:29.951242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tf4\" (UniqueName: \"kubernetes.io/projected/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-kube-api-access-b8tf4\") pod \"payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx\" (UID: \"87d9bde0-b1b2-45f5-a154-ea0a584e66a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:20:30.052172 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.052131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfwrq\" (UniqueName: \"kubernetes.io/projected/a3b7cf90-2215-402f-acbf-710f216ac467-kube-api-access-zfwrq\") pod \"payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776\" (UID: \"a3b7cf90-2215-402f-acbf-710f216ac467\") " pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:20:30.052172 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.052172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx\" (UID: \"87d9bde0-b1b2-45f5-a154-ea0a584e66a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:20:30.052380 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.052225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b7cf90-2215-402f-acbf-710f216ac467-plugins-config-volume\") pod \"payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776\" (UID: \"a3b7cf90-2215-402f-acbf-710f216ac467\") " pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:20:30.052380 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.052247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tf4\" (UniqueName: \"kubernetes.io/projected/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-kube-api-access-b8tf4\") pod \"payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx\" (UID: \"87d9bde0-b1b2-45f5-a154-ea0a584e66a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:20:30.053042 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.053020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b7cf90-2215-402f-acbf-710f216ac467-plugins-config-volume\") pod \"payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776\" (UID: \"a3b7cf90-2215-402f-acbf-710f216ac467\") " pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:20:30.053092 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.053030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx\" (UID: \"87d9bde0-b1b2-45f5-a154-ea0a584e66a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:20:30.060807 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.060749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tf4\" (UniqueName: \"kubernetes.io/projected/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-kube-api-access-b8tf4\") pod \"payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx\" (UID: \"87d9bde0-b1b2-45f5-a154-ea0a584e66a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:20:30.060807 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.060790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfwrq\" (UniqueName: \"kubernetes.io/projected/a3b7cf90-2215-402f-acbf-710f216ac467-kube-api-access-zfwrq\") pod \"payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776\" (UID: \"a3b7cf90-2215-402f-acbf-710f216ac467\") " pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:20:30.123384 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.123354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:20:30.169098 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.169054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:20:30.393942 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.393920    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"]
Sep 09 03:20:30.396341 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:20:30.396316    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b7cf90_2215_402f_acbf_710f216ac467.slice/crio-7c6785ae2aede68347b7f3de9548686e85fe2483eecf5b488d63b1dada151334 WatchSource:0}: Error finding container 7c6785ae2aede68347b7f3de9548686e85fe2483eecf5b488d63b1dada151334: Status 404 returned error can't find the container with id 7c6785ae2aede68347b7f3de9548686e85fe2483eecf5b488d63b1dada151334
Sep 09 03:20:30.454174 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.454141    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"]
Sep 09 03:20:30.457010 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:20:30.456990    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d9bde0_b1b2_45f5_a154_ea0a584e66a4.slice/crio-22a41eb8e15467aedc7029d98b2f4b4cacf1facc4268d0da175f2cd7dea2c3bb WatchSource:0}: Error finding container 22a41eb8e15467aedc7029d98b2f4b4cacf1facc4268d0da175f2cd7dea2c3bb: Status 404 returned error can't find the container with id 22a41eb8e15467aedc7029d98b2f4b4cacf1facc4268d0da175f2cd7dea2c3bb
Sep 09 03:20:30.534713 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.534676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx" event={"ID":"87d9bde0-b1b2-45f5-a154-ea0a584e66a4","Type":"ContainerStarted","Data":"e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed"}
Sep 09 03:20:30.534919 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.534861    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:20:30.534919 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.534893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx" event={"ID":"87d9bde0-b1b2-45f5-a154-ea0a584e66a4","Type":"ContainerStarted","Data":"22a41eb8e15467aedc7029d98b2f4b4cacf1facc4268d0da175f2cd7dea2c3bb"}
Sep 09 03:20:30.536494 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.536463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776" event={"ID":"a3b7cf90-2215-402f-acbf-710f216ac467","Type":"ContainerStarted","Data":"47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363"}
Sep 09 03:20:30.536644 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.536525    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:20:30.536644 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.536551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776" event={"ID":"a3b7cf90-2215-402f-acbf-710f216ac467","Type":"ContainerStarted","Data":"7c6785ae2aede68347b7f3de9548686e85fe2483eecf5b488d63b1dada151334"}
Sep 09 03:20:30.554691 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.554647    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx" podStartSLOduration=1.554633003 podStartE2EDuration="1.554633003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:30.553347036 +0000 UTC m=+874.079661810" watchObservedRunningTime="2026-09-09 03:20:30.554633003 +0000 UTC m=+874.080947742"
Sep 09 03:20:30.576602 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.576522    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776" podStartSLOduration=1.5765031010000001 podStartE2EDuration="1.576503101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:30.574492976 +0000 UTC m=+874.100807741" watchObservedRunningTime="2026-09-09 03:20:30.576503101 +0000 UTC m=+874.102817841"
Sep 09 03:20:30.721334 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.721255    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"]
Sep 09 03:20:30.723796 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.723779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:20:30.725907 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.725887    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-be414f-dockercfg-5zgfm\""
Sep 09 03:20:30.726018 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.725947    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-be414f\""
Sep 09 03:20:30.734459 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.734426    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"]
Sep 09 03:20:30.737091 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.737066    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"]
Sep 09 03:20:30.739457 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.739432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:20:30.764116 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.764088    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"]
Sep 09 03:20:30.865673 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.865635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af89f475-897b-46aa-9e30-1d0051712b25-plugins-config-volume\") pod \"payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n\" (UID: \"af89f475-897b-46aa-9e30-1d0051712b25\") " pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:20:30.866172 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.865684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sdd9\" (UniqueName: \"kubernetes.io/projected/7685bde2-1032-456a-87ab-cbcd4310b27d-kube-api-access-9sdd9\") pod \"payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz\" (UID: \"7685bde2-1032-456a-87ab-cbcd4310b27d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:20:30.866172 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.865812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7685bde2-1032-456a-87ab-cbcd4310b27d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz\" (UID: \"7685bde2-1032-456a-87ab-cbcd4310b27d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:20:30.866172 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.865876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m6w8\" (UniqueName: \"kubernetes.io/projected/af89f475-897b-46aa-9e30-1d0051712b25-kube-api-access-4m6w8\") pod \"payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n\" (UID: \"af89f475-897b-46aa-9e30-1d0051712b25\") " pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:20:30.967132 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.967088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7685bde2-1032-456a-87ab-cbcd4310b27d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz\" (UID: \"7685bde2-1032-456a-87ab-cbcd4310b27d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:20:30.967323 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.967143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6w8\" (UniqueName: \"kubernetes.io/projected/af89f475-897b-46aa-9e30-1d0051712b25-kube-api-access-4m6w8\") pod \"payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n\" (UID: \"af89f475-897b-46aa-9e30-1d0051712b25\") " pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:20:30.967323 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.967223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af89f475-897b-46aa-9e30-1d0051712b25-plugins-config-volume\") pod \"payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n\" (UID: \"af89f475-897b-46aa-9e30-1d0051712b25\") " pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:20:30.967323 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.967260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdd9\" (UniqueName: \"kubernetes.io/projected/7685bde2-1032-456a-87ab-cbcd4310b27d-kube-api-access-9sdd9\") pod \"payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz\" (UID: \"7685bde2-1032-456a-87ab-cbcd4310b27d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:20:30.968012 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.967986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af89f475-897b-46aa-9e30-1d0051712b25-plugins-config-volume\") pod \"payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n\" (UID: \"af89f475-897b-46aa-9e30-1d0051712b25\") " pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:20:30.968117 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.968040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7685bde2-1032-456a-87ab-cbcd4310b27d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz\" (UID: \"7685bde2-1032-456a-87ab-cbcd4310b27d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:20:30.974674 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.974617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m6w8\" (UniqueName: \"kubernetes.io/projected/af89f475-897b-46aa-9e30-1d0051712b25-kube-api-access-4m6w8\") pod \"payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n\" (UID: \"af89f475-897b-46aa-9e30-1d0051712b25\") " pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:20:30.974789 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:30.974766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sdd9\" (UniqueName: \"kubernetes.io/projected/7685bde2-1032-456a-87ab-cbcd4310b27d-kube-api-access-9sdd9\") pod \"payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz\" (UID: \"7685bde2-1032-456a-87ab-cbcd4310b27d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:20:31.035161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:31.035127    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:20:31.061334 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:31.061304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:20:31.314030 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:31.313957    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"]
Sep 09 03:20:31.317107 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:20:31.317082    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7685bde2_1032_456a_87ab_cbcd4310b27d.slice/crio-ba64141ff3d7aad76af8d091b4477bebf03cc9f5227198683c836a0bd5af4bd8 WatchSource:0}: Error finding container ba64141ff3d7aad76af8d091b4477bebf03cc9f5227198683c836a0bd5af4bd8: Status 404 returned error can't find the container with id ba64141ff3d7aad76af8d091b4477bebf03cc9f5227198683c836a0bd5af4bd8
Sep 09 03:20:31.345066 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:31.345043    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"]
Sep 09 03:20:31.354705 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:20:31.354676    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf89f475_897b_46aa_9e30_1d0051712b25.slice/crio-cb7c34807f4c09b4348ab30276f91ea576ac4fd76660598ec8e7de14132a7bd0 WatchSource:0}: Error finding container cb7c34807f4c09b4348ab30276f91ea576ac4fd76660598ec8e7de14132a7bd0: Status 404 returned error can't find the container with id cb7c34807f4c09b4348ab30276f91ea576ac4fd76660598ec8e7de14132a7bd0
Sep 09 03:20:31.544500 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:31.544462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n" event={"ID":"af89f475-897b-46aa-9e30-1d0051712b25","Type":"ContainerStarted","Data":"dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88"}
Sep 09 03:20:31.544717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:31.544525    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:20:31.544717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:31.544547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n" event={"ID":"af89f475-897b-46aa-9e30-1d0051712b25","Type":"ContainerStarted","Data":"cb7c34807f4c09b4348ab30276f91ea576ac4fd76660598ec8e7de14132a7bd0"}
Sep 09 03:20:31.546185 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:31.546160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz" event={"ID":"7685bde2-1032-456a-87ab-cbcd4310b27d","Type":"ContainerStarted","Data":"950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17"}
Sep 09 03:20:31.546320 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:31.546203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz" event={"ID":"7685bde2-1032-456a-87ab-cbcd4310b27d","Type":"ContainerStarted","Data":"ba64141ff3d7aad76af8d091b4477bebf03cc9f5227198683c836a0bd5af4bd8"}
Sep 09 03:20:31.561259 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:31.561218    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n" podStartSLOduration=1.561204107 podStartE2EDuration="1.561204107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:31.559809464 +0000 UTC m=+875.086124220" watchObservedRunningTime="2026-09-09 03:20:31.561204107 +0000 UTC m=+875.087518845"
Sep 09 03:20:31.576695 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:31.576619    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz" podStartSLOduration=1.5766039360000002 podStartE2EDuration="1.576603936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:31.575731374 +0000 UTC m=+875.102046113" watchObservedRunningTime="2026-09-09 03:20:31.576603936 +0000 UTC m=+875.102918678"
Sep 09 03:20:32.553121 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:32.553093    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:20:34.505560 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:34.505530    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:20:34.505942 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:34.505638    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:20:36.013451 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:36.013414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:20:41.548349 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:41.548317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:20:41.548806 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:41.548403    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:20:42.555172 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:42.555142    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:20:43.561018 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:43.560989    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:20:50.709231 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.709196    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"]
Sep 09 03:20:50.712557 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.712542    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:20:50.714923 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.714902    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-70784b-dockercfg-v8v64\""
Sep 09 03:20:50.715022 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.714907    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-70784b\""
Sep 09 03:20:50.722639 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.722611    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"]
Sep 09 03:20:50.743326 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.743304    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"]
Sep 09 03:20:50.745556 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.745540    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:20:50.753679 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.750106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbfx6\" (UniqueName: \"kubernetes.io/projected/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-kube-api-access-cbfx6\") pod \"payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw\" (UID: \"bd22869f-85a0-40e6-b9d8-55b21ae1da6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:20:50.753679 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.750197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw\" (UID: \"bd22869f-85a0-40e6-b9d8-55b21ae1da6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:20:50.758546 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.758517    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"]
Sep 09 03:20:50.851505 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.851468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbfx6\" (UniqueName: \"kubernetes.io/projected/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-kube-api-access-cbfx6\") pod \"payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw\" (UID: \"bd22869f-85a0-40e6-b9d8-55b21ae1da6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:20:50.851676 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.851517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj\" (UID: \"f5173d2d-efe2-42b8-b25c-b2f0b5c46857\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:20:50.851676 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.851556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw\" (UID: \"bd22869f-85a0-40e6-b9d8-55b21ae1da6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:20:50.851676 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.851619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4844c\" (UniqueName: \"kubernetes.io/projected/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-kube-api-access-4844c\") pod \"payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj\" (UID: \"f5173d2d-efe2-42b8-b25c-b2f0b5c46857\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:20:50.852237 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.852217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw\" (UID: \"bd22869f-85a0-40e6-b9d8-55b21ae1da6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:20:50.859845 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.859823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbfx6\" (UniqueName: \"kubernetes.io/projected/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-kube-api-access-cbfx6\") pod \"payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw\" (UID: \"bd22869f-85a0-40e6-b9d8-55b21ae1da6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:20:50.952379 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.952342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4844c\" (UniqueName: \"kubernetes.io/projected/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-kube-api-access-4844c\") pod \"payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj\" (UID: \"f5173d2d-efe2-42b8-b25c-b2f0b5c46857\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:20:50.952568 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.952423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj\" (UID: \"f5173d2d-efe2-42b8-b25c-b2f0b5c46857\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:20:50.953085 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.953065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj\" (UID: \"f5173d2d-efe2-42b8-b25c-b2f0b5c46857\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:20:50.963256 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:50.963185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4844c\" (UniqueName: \"kubernetes.io/projected/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-kube-api-access-4844c\") pod \"payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj\" (UID: \"f5173d2d-efe2-42b8-b25c-b2f0b5c46857\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:20:51.023880 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.023853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:20:51.062456 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.062426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:20:51.310822 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.310797    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"]
Sep 09 03:20:51.312469 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:20:51.312444    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd22869f_85a0_40e6_b9d8_55b21ae1da6b.slice/crio-9dbabbf7b345ea61016fa97c5839dd96f841f105e342acce593342b04cb1db4a WatchSource:0}: Error finding container 9dbabbf7b345ea61016fa97c5839dd96f841f105e342acce593342b04cb1db4a: Status 404 returned error can't find the container with id 9dbabbf7b345ea61016fa97c5839dd96f841f105e342acce593342b04cb1db4a
Sep 09 03:20:51.349734 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.349711    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"]
Sep 09 03:20:51.351815 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:20:51.351789    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5173d2d_efe2_42b8_b25c_b2f0b5c46857.slice/crio-43d7ebbb7e2cdf981a7448136a88aaf0be71adaa866314214dc37939369ea779 WatchSource:0}: Error finding container 43d7ebbb7e2cdf981a7448136a88aaf0be71adaa866314214dc37939369ea779: Status 404 returned error can't find the container with id 43d7ebbb7e2cdf981a7448136a88aaf0be71adaa866314214dc37939369ea779
Sep 09 03:20:51.649979 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.649942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj" event={"ID":"f5173d2d-efe2-42b8-b25c-b2f0b5c46857","Type":"ContainerStarted","Data":"a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1"}
Sep 09 03:20:51.650205 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.650013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:20:51.650205 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.650039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj" event={"ID":"f5173d2d-efe2-42b8-b25c-b2f0b5c46857","Type":"ContainerStarted","Data":"43d7ebbb7e2cdf981a7448136a88aaf0be71adaa866314214dc37939369ea779"}
Sep 09 03:20:51.651649 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.651621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw" event={"ID":"bd22869f-85a0-40e6-b9d8-55b21ae1da6b","Type":"ContainerStarted","Data":"e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8"}
Sep 09 03:20:51.651786 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.651668    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:20:51.651786 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.651691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw" event={"ID":"bd22869f-85a0-40e6-b9d8-55b21ae1da6b","Type":"ContainerStarted","Data":"9dbabbf7b345ea61016fa97c5839dd96f841f105e342acce593342b04cb1db4a"}
Sep 09 03:20:51.668587 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.668520    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj" podStartSLOduration=1.6685042559999999 podStartE2EDuration="1.668504256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:51.666915124 +0000 UTC m=+895.193229863" watchObservedRunningTime="2026-09-09 03:20:51.668504256 +0000 UTC m=+895.194818995"
Sep 09 03:20:51.690532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:51.690491    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw" podStartSLOduration=1.690474601 podStartE2EDuration="1.690474601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:51.688170458 +0000 UTC m=+895.214485217" watchObservedRunningTime="2026-09-09 03:20:51.690474601 +0000 UTC m=+895.216789340"
Sep 09 03:20:55.317898 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:20:55.317866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:21:00.035671 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.035632    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"]
Sep 09 03:21:00.036067 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.035933    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776" podUID="a3b7cf90-2215-402f-acbf-710f216ac467" containerName="payload-processing" containerID="cri-o://47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363" gracePeriod=30
Sep 09 03:21:00.044687 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.044656    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"]
Sep 09 03:21:00.044959 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.044892    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx" podUID="87d9bde0-b1b2-45f5-a154-ea0a584e66a4" containerName="payload-pre-processing" containerID="cri-o://e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed" gracePeriod=30
Sep 09 03:21:00.309995 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.309970    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:21:00.313719 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.313687    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:21:00.440462 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.440426    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-plugins-config-volume\") pod \"87d9bde0-b1b2-45f5-a154-ea0a584e66a4\" (UID: \"87d9bde0-b1b2-45f5-a154-ea0a584e66a4\") "
Sep 09 03:21:00.440643 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.440500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfwrq\" (UniqueName: \"kubernetes.io/projected/a3b7cf90-2215-402f-acbf-710f216ac467-kube-api-access-zfwrq\") pod \"a3b7cf90-2215-402f-acbf-710f216ac467\" (UID: \"a3b7cf90-2215-402f-acbf-710f216ac467\") "
Sep 09 03:21:00.440643 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.440529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8tf4\" (UniqueName: \"kubernetes.io/projected/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-kube-api-access-b8tf4\") pod \"87d9bde0-b1b2-45f5-a154-ea0a584e66a4\" (UID: \"87d9bde0-b1b2-45f5-a154-ea0a584e66a4\") "
Sep 09 03:21:00.440643 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.440629    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b7cf90-2215-402f-acbf-710f216ac467-plugins-config-volume\") pod \"a3b7cf90-2215-402f-acbf-710f216ac467\" (UID: \"a3b7cf90-2215-402f-acbf-710f216ac467\") "
Sep 09 03:21:00.440975 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.440898    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87d9bde0-b1b2-45f5-a154-ea0a584e66a4" (UID: "87d9bde0-b1b2-45f5-a154-ea0a584e66a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:21:00.441108 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.441050    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b7cf90-2215-402f-acbf-710f216ac467-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3b7cf90-2215-402f-acbf-710f216ac467" (UID: "a3b7cf90-2215-402f-acbf-710f216ac467"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:21:00.443040 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.443016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-kube-api-access-b8tf4" (OuterVolumeSpecName: "kube-api-access-b8tf4") pod "87d9bde0-b1b2-45f5-a154-ea0a584e66a4" (UID: "87d9bde0-b1b2-45f5-a154-ea0a584e66a4"). InnerVolumeSpecName "kube-api-access-b8tf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:21:00.443129 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.443078    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b7cf90-2215-402f-acbf-710f216ac467-kube-api-access-zfwrq" (OuterVolumeSpecName: "kube-api-access-zfwrq") pod "a3b7cf90-2215-402f-acbf-710f216ac467" (UID: "a3b7cf90-2215-402f-acbf-710f216ac467"). InnerVolumeSpecName "kube-api-access-zfwrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:21:00.541826 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.541738    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b7cf90-2215-402f-acbf-710f216ac467-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:21:00.541826 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.541773    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:21:00.541826 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.541782    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfwrq\" (UniqueName: \"kubernetes.io/projected/a3b7cf90-2215-402f-acbf-710f216ac467-kube-api-access-zfwrq\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:21:00.541826 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.541791    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8tf4\" (UniqueName: \"kubernetes.io/projected/87d9bde0-b1b2-45f5-a154-ea0a584e66a4-kube-api-access-b8tf4\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:21:00.698112 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.698069    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3b7cf90-2215-402f-acbf-710f216ac467" containerID="47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363" exitCode=0
Sep 09 03:21:00.698307 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.698128    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"
Sep 09 03:21:00.698307 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.698156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776" event={"ID":"a3b7cf90-2215-402f-acbf-710f216ac467","Type":"ContainerDied","Data":"47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363"}
Sep 09 03:21:00.698307 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.698246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776" event={"ID":"a3b7cf90-2215-402f-acbf-710f216ac467","Type":"ContainerDied","Data":"7c6785ae2aede68347b7f3de9548686e85fe2483eecf5b488d63b1dada151334"}
Sep 09 03:21:00.698307 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.698278    2580 scope.go:117] "RemoveContainer" containerID="47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363"
Sep 09 03:21:00.699672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.699642    2580 generic.go:356] "Generic (PLEG): container finished" podID="87d9bde0-b1b2-45f5-a154-ea0a584e66a4" containerID="e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed" exitCode=0
Sep 09 03:21:00.699777 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.699675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx" event={"ID":"87d9bde0-b1b2-45f5-a154-ea0a584e66a4","Type":"ContainerDied","Data":"e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed"}
Sep 09 03:21:00.699777 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.699699    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"
Sep 09 03:21:00.699777 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.699728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx" event={"ID":"87d9bde0-b1b2-45f5-a154-ea0a584e66a4","Type":"ContainerDied","Data":"22a41eb8e15467aedc7029d98b2f4b4cacf1facc4268d0da175f2cd7dea2c3bb"}
Sep 09 03:21:00.708172 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.708150    2580 scope.go:117] "RemoveContainer" containerID="47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363"
Sep 09 03:21:00.708486 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:21:00.708465    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363\": container with ID starting with 47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363 not found: ID does not exist" containerID="47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363"
Sep 09 03:21:00.708602 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.708491    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363"} err="failed to get container status \"47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363\": rpc error: code = NotFound desc = could not find container \"47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363\": container with ID starting with 47f6b8f516c4a19b29aa673d19c27fa5c3abcbc96dc997ad86c1c10f0148e363 not found: ID does not exist"
Sep 09 03:21:00.708602 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.708507    2580 scope.go:117] "RemoveContainer" containerID="e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed"
Sep 09 03:21:00.716701 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.716682    2580 scope.go:117] "RemoveContainer" containerID="e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed"
Sep 09 03:21:00.716958 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:21:00.716938    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed\": container with ID starting with e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed not found: ID does not exist" containerID="e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed"
Sep 09 03:21:00.717020 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.716963    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed"} err="failed to get container status \"e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed\": rpc error: code = NotFound desc = could not find container \"e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed\": container with ID starting with e61c2be7fa3b06ad05927d2a81cb7ec0fd2e016b656863da7f85f4c2eee9f0ed not found: ID does not exist"
Sep 09 03:21:00.723084 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.723066    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"]
Sep 09 03:21:00.727308 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.727289    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ecf63f7-54c96c7494-dgdmx"]
Sep 09 03:21:00.741001 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.740976    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"]
Sep 09 03:21:00.748990 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.748966    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ecf63f7-6f4f49c95c-n4776"]
Sep 09 03:21:00.971698 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.971667    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d9bde0-b1b2-45f5-a154-ea0a584e66a4" path="/var/lib/kubelet/pods/87d9bde0-b1b2-45f5-a154-ea0a584e66a4/volumes"
Sep 09 03:21:00.972061 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:00.972048    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3b7cf90-2215-402f-acbf-710f216ac467" path="/var/lib/kubelet/pods/a3b7cf90-2215-402f-acbf-710f216ac467/volumes"
Sep 09 03:21:02.659772 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:02.659741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:21:02.660260 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:02.660089    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:21:05.248378 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.248339    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"]
Sep 09 03:21:05.248775 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.248761    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3b7cf90-2215-402f-acbf-710f216ac467" containerName="payload-processing"
Sep 09 03:21:05.248819 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.248777    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3b7cf90-2215-402f-acbf-710f216ac467" containerName="payload-processing"
Sep 09 03:21:05.248819 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.248791    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d9bde0-b1b2-45f5-a154-ea0a584e66a4" containerName="payload-pre-processing"
Sep 09 03:21:05.248819 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.248797    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d9bde0-b1b2-45f5-a154-ea0a584e66a4" containerName="payload-pre-processing"
Sep 09 03:21:05.248903 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.248856    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d9bde0-b1b2-45f5-a154-ea0a584e66a4" containerName="payload-pre-processing"
Sep 09 03:21:05.248903 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.248864    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3b7cf90-2215-402f-acbf-710f216ac467" containerName="payload-processing"
Sep 09 03:21:05.253668 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.253648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:05.257430 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.257406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-0e6f9624-dockercfg-7ckvs\""
Sep 09 03:21:05.257694 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.257678    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0e6f9624\""
Sep 09 03:21:05.267676 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.267657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"]
Sep 09 03:21:05.278175 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.278142    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"]
Sep 09 03:21:05.282752 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.282730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:05.293896 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.293874    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"]
Sep 09 03:21:05.385179 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.385143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c69e61-0741-4c1e-aea8-2bb73a7b9594-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn\" (UID: \"87c69e61-0741-4c1e-aea8-2bb73a7b9594\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:05.385346 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.385190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c57d68-5534-47f4-8143-fc59c3fdb53d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv\" (UID: \"e5c57d68-5534-47f4-8143-fc59c3fdb53d\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:05.385346 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.385218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkssf\" (UniqueName: \"kubernetes.io/projected/e5c57d68-5534-47f4-8143-fc59c3fdb53d-kube-api-access-xkssf\") pod \"payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv\" (UID: \"e5c57d68-5534-47f4-8143-fc59c3fdb53d\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:05.385418 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.385349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh2xp\" (UniqueName: \"kubernetes.io/projected/87c69e61-0741-4c1e-aea8-2bb73a7b9594-kube-api-access-hh2xp\") pod \"payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn\" (UID: \"87c69e61-0741-4c1e-aea8-2bb73a7b9594\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:05.486816 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.486763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkssf\" (UniqueName: \"kubernetes.io/projected/e5c57d68-5534-47f4-8143-fc59c3fdb53d-kube-api-access-xkssf\") pod \"payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv\" (UID: \"e5c57d68-5534-47f4-8143-fc59c3fdb53d\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:05.487128 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.486937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh2xp\" (UniqueName: \"kubernetes.io/projected/87c69e61-0741-4c1e-aea8-2bb73a7b9594-kube-api-access-hh2xp\") pod \"payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn\" (UID: \"87c69e61-0741-4c1e-aea8-2bb73a7b9594\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:05.487128 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.486992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c69e61-0741-4c1e-aea8-2bb73a7b9594-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn\" (UID: \"87c69e61-0741-4c1e-aea8-2bb73a7b9594\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:05.487128 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.487029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c57d68-5534-47f4-8143-fc59c3fdb53d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv\" (UID: \"e5c57d68-5534-47f4-8143-fc59c3fdb53d\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:05.487904 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.487882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c57d68-5534-47f4-8143-fc59c3fdb53d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv\" (UID: \"e5c57d68-5534-47f4-8143-fc59c3fdb53d\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:05.487996 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.487909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c69e61-0741-4c1e-aea8-2bb73a7b9594-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn\" (UID: \"87c69e61-0741-4c1e-aea8-2bb73a7b9594\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:05.495899 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.495880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh2xp\" (UniqueName: \"kubernetes.io/projected/87c69e61-0741-4c1e-aea8-2bb73a7b9594-kube-api-access-hh2xp\") pod \"payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn\" (UID: \"87c69e61-0741-4c1e-aea8-2bb73a7b9594\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:05.495977 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.495939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkssf\" (UniqueName: \"kubernetes.io/projected/e5c57d68-5534-47f4-8143-fc59c3fdb53d-kube-api-access-xkssf\") pod \"payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv\" (UID: \"e5c57d68-5534-47f4-8143-fc59c3fdb53d\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:05.563359 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.563292    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:05.594323 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.594278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:05.849893 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.849867    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"]
Sep 09 03:21:05.852378 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:21:05.852354    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c69e61_0741_4c1e_aea8_2bb73a7b9594.slice/crio-eefef8a810af5b18cce2469e7f1650f3972774e927fc34433a149214bffb43f7 WatchSource:0}: Error finding container eefef8a810af5b18cce2469e7f1650f3972774e927fc34433a149214bffb43f7: Status 404 returned error can't find the container with id eefef8a810af5b18cce2469e7f1650f3972774e927fc34433a149214bffb43f7
Sep 09 03:21:05.883034 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:05.883010    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"]
Sep 09 03:21:05.884937 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:21:05.884907    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c57d68_5534_47f4_8143_fc59c3fdb53d.slice/crio-df0caeabe5d26267cd2cd16d28576f431ac4a36e4d0a9f55645a0d68dd466a68 WatchSource:0}: Error finding container df0caeabe5d26267cd2cd16d28576f431ac4a36e4d0a9f55645a0d68dd466a68: Status 404 returned error can't find the container with id df0caeabe5d26267cd2cd16d28576f431ac4a36e4d0a9f55645a0d68dd466a68
Sep 09 03:21:06.734186 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:06.734152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn" event={"ID":"87c69e61-0741-4c1e-aea8-2bb73a7b9594","Type":"ContainerStarted","Data":"6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a"}
Sep 09 03:21:06.734699 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:06.734212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn" event={"ID":"87c69e61-0741-4c1e-aea8-2bb73a7b9594","Type":"ContainerStarted","Data":"eefef8a810af5b18cce2469e7f1650f3972774e927fc34433a149214bffb43f7"}
Sep 09 03:21:06.734699 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:06.734238    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:06.735906 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:06.735884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv" event={"ID":"e5c57d68-5534-47f4-8143-fc59c3fdb53d","Type":"ContainerStarted","Data":"c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c"}
Sep 09 03:21:06.736023 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:06.735920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv" event={"ID":"e5c57d68-5534-47f4-8143-fc59c3fdb53d","Type":"ContainerStarted","Data":"df0caeabe5d26267cd2cd16d28576f431ac4a36e4d0a9f55645a0d68dd466a68"}
Sep 09 03:21:06.736023 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:06.736009    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:06.762694 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:06.762633    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn" podStartSLOduration=1.762612956 podStartE2EDuration="1.762612956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:06.760258834 +0000 UTC m=+910.286573598" watchObservedRunningTime="2026-09-09 03:21:06.762612956 +0000 UTC m=+910.288927698"
Sep 09 03:21:06.781826 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:06.781765    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv" podStartSLOduration=1.7817450639999999 podStartE2EDuration="1.781745064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:06.780056982 +0000 UTC m=+910.306371724" watchObservedRunningTime="2026-09-09 03:21:06.781745064 +0000 UTC m=+910.308059804"
Sep 09 03:21:13.608279 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:13.608241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:21:17.744134 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:17.744099    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:17.746439 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:17.744272    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:28.600299 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:28.600263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:21:35.267356 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.267318    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"]
Sep 09 03:21:35.267761 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.267609    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv" podUID="e5c57d68-5534-47f4-8143-fc59c3fdb53d" containerName="payload-processing" containerID="cri-o://c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c" gracePeriod=30
Sep 09 03:21:35.277751 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.277716    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"]
Sep 09 03:21:35.277997 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.277959    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn" podUID="87c69e61-0741-4c1e-aea8-2bb73a7b9594" containerName="payload-pre-processing" containerID="cri-o://6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a" gracePeriod=30
Sep 09 03:21:35.561553 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.561521    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:35.565664 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.565623    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:35.657058 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.657010    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh2xp\" (UniqueName: \"kubernetes.io/projected/87c69e61-0741-4c1e-aea8-2bb73a7b9594-kube-api-access-hh2xp\") pod \"87c69e61-0741-4c1e-aea8-2bb73a7b9594\" (UID: \"87c69e61-0741-4c1e-aea8-2bb73a7b9594\") "
Sep 09 03:21:35.657260 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.657081    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c69e61-0741-4c1e-aea8-2bb73a7b9594-plugins-config-volume\") pod \"87c69e61-0741-4c1e-aea8-2bb73a7b9594\" (UID: \"87c69e61-0741-4c1e-aea8-2bb73a7b9594\") "
Sep 09 03:21:35.657260 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.657104    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkssf\" (UniqueName: \"kubernetes.io/projected/e5c57d68-5534-47f4-8143-fc59c3fdb53d-kube-api-access-xkssf\") pod \"e5c57d68-5534-47f4-8143-fc59c3fdb53d\" (UID: \"e5c57d68-5534-47f4-8143-fc59c3fdb53d\") "
Sep 09 03:21:35.657260 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.657194    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c57d68-5534-47f4-8143-fc59c3fdb53d-plugins-config-volume\") pod \"e5c57d68-5534-47f4-8143-fc59c3fdb53d\" (UID: \"e5c57d68-5534-47f4-8143-fc59c3fdb53d\") "
Sep 09 03:21:35.659561 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.657846    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c69e61-0741-4c1e-aea8-2bb73a7b9594-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87c69e61-0741-4c1e-aea8-2bb73a7b9594" (UID: "87c69e61-0741-4c1e-aea8-2bb73a7b9594"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:21:35.659561 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.657851    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c57d68-5534-47f4-8143-fc59c3fdb53d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5c57d68-5534-47f4-8143-fc59c3fdb53d" (UID: "e5c57d68-5534-47f4-8143-fc59c3fdb53d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:21:35.660886 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.660855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c57d68-5534-47f4-8143-fc59c3fdb53d-kube-api-access-xkssf" (OuterVolumeSpecName: "kube-api-access-xkssf") pod "e5c57d68-5534-47f4-8143-fc59c3fdb53d" (UID: "e5c57d68-5534-47f4-8143-fc59c3fdb53d"). InnerVolumeSpecName "kube-api-access-xkssf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:21:35.661398 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.661366    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c69e61-0741-4c1e-aea8-2bb73a7b9594-kube-api-access-hh2xp" (OuterVolumeSpecName: "kube-api-access-hh2xp") pod "87c69e61-0741-4c1e-aea8-2bb73a7b9594" (UID: "87c69e61-0741-4c1e-aea8-2bb73a7b9594"). InnerVolumeSpecName "kube-api-access-hh2xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:21:35.758823 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.758787    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c57d68-5534-47f4-8143-fc59c3fdb53d-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:21:35.758823 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.758816    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh2xp\" (UniqueName: \"kubernetes.io/projected/87c69e61-0741-4c1e-aea8-2bb73a7b9594-kube-api-access-hh2xp\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:21:35.758823 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.758826    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c69e61-0741-4c1e-aea8-2bb73a7b9594-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:21:35.759081 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.758836    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkssf\" (UniqueName: \"kubernetes.io/projected/e5c57d68-5534-47f4-8143-fc59c3fdb53d-kube-api-access-xkssf\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:21:35.883058 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.883021    2580 generic.go:356] "Generic (PLEG): container finished" podID="87c69e61-0741-4c1e-aea8-2bb73a7b9594" containerID="6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a" exitCode=0
Sep 09 03:21:35.883271 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.883070    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"
Sep 09 03:21:35.883271 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.883114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn" event={"ID":"87c69e61-0741-4c1e-aea8-2bb73a7b9594","Type":"ContainerDied","Data":"6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a"}
Sep 09 03:21:35.883271 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.883173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn" event={"ID":"87c69e61-0741-4c1e-aea8-2bb73a7b9594","Type":"ContainerDied","Data":"eefef8a810af5b18cce2469e7f1650f3972774e927fc34433a149214bffb43f7"}
Sep 09 03:21:35.883271 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.883205    2580 scope.go:117] "RemoveContainer" containerID="6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a"
Sep 09 03:21:35.884649 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.884630    2580 generic.go:356] "Generic (PLEG): container finished" podID="e5c57d68-5534-47f4-8143-fc59c3fdb53d" containerID="c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c" exitCode=0
Sep 09 03:21:35.884753 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.884676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv" event={"ID":"e5c57d68-5534-47f4-8143-fc59c3fdb53d","Type":"ContainerDied","Data":"c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c"}
Sep 09 03:21:35.884753 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.884695    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"
Sep 09 03:21:35.884753 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.884710    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv" event={"ID":"e5c57d68-5534-47f4-8143-fc59c3fdb53d","Type":"ContainerDied","Data":"df0caeabe5d26267cd2cd16d28576f431ac4a36e4d0a9f55645a0d68dd466a68"}
Sep 09 03:21:35.894210 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.894185    2580 scope.go:117] "RemoveContainer" containerID="6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a"
Sep 09 03:21:35.894515 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:21:35.894497    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a\": container with ID starting with 6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a not found: ID does not exist" containerID="6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a"
Sep 09 03:21:35.894626 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.894525    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a"} err="failed to get container status \"6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a\": rpc error: code = NotFound desc = could not find container \"6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a\": container with ID starting with 6d84b84536805b8fe57ecf449e9e899128224322e01818ff47450c06f8f1d71a not found: ID does not exist"
Sep 09 03:21:35.894626 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.894541    2580 scope.go:117] "RemoveContainer" containerID="c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c"
Sep 09 03:21:35.902981 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.902965    2580 scope.go:117] "RemoveContainer" containerID="c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c"
Sep 09 03:21:35.903280 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:21:35.903260    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c\": container with ID starting with c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c not found: ID does not exist" containerID="c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c"
Sep 09 03:21:35.903346 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.903286    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c"} err="failed to get container status \"c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c\": rpc error: code = NotFound desc = could not find container \"c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c\": container with ID starting with c01177a17dbe520bd7a5492e307889181794156c8bc579f6450e196f5cdfda7c not found: ID does not exist"
Sep 09 03:21:35.908095 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.908068    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"]
Sep 09 03:21:35.913567 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.913535    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0e6f9624-789949854b-j2rfn"]
Sep 09 03:21:35.926360 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.926334    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"]
Sep 09 03:21:35.930717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:35.930695    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e6f9624-7bd5fc7b46-g6rhv"]
Sep 09 03:21:36.971009 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:36.970977    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87c69e61-0741-4c1e-aea8-2bb73a7b9594" path="/var/lib/kubelet/pods/87c69e61-0741-4c1e-aea8-2bb73a7b9594/volumes"
Sep 09 03:21:36.971398 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:36.971315    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5c57d68-5534-47f4-8143-fc59c3fdb53d" path="/var/lib/kubelet/pods/e5c57d68-5534-47f4-8143-fc59c3fdb53d/volumes"
Sep 09 03:21:41.095216 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.095180    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"]
Sep 09 03:21:41.095768 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.095669    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87c69e61-0741-4c1e-aea8-2bb73a7b9594" containerName="payload-pre-processing"
Sep 09 03:21:41.095768 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.095684    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87c69e61-0741-4c1e-aea8-2bb73a7b9594" containerName="payload-pre-processing"
Sep 09 03:21:41.095768 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.095700    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c57d68-5534-47f4-8143-fc59c3fdb53d" containerName="payload-processing"
Sep 09 03:21:41.095768 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.095708    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c57d68-5534-47f4-8143-fc59c3fdb53d" containerName="payload-processing"
Sep 09 03:21:41.096555 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.096528    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87c69e61-0741-4c1e-aea8-2bb73a7b9594" containerName="payload-pre-processing"
Sep 09 03:21:41.096686 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.096600    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c57d68-5534-47f4-8143-fc59c3fdb53d" containerName="payload-processing"
Sep 09 03:21:41.105031 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.105009    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:21:41.107953 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.107926    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-189a872e\""
Sep 09 03:21:41.108297 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.108280    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-189a872e-dockercfg-bz5m6\""
Sep 09 03:21:41.109200 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.109179    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"]
Sep 09 03:21:41.127993 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.127959    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"]
Sep 09 03:21:41.131374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.131354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:21:41.145989 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.145959    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"]
Sep 09 03:21:41.211508 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.211469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlc5n\" (UniqueName: \"kubernetes.io/projected/4300c43d-6031-48b8-a4b5-868df90b4700-kube-api-access-nlc5n\") pod \"payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd\" (UID: \"4300c43d-6031-48b8-a4b5-868df90b4700\") " pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:21:41.211508 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.211509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20086477-254d-49c8-83b8-9534c5fb88a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv\" (UID: \"20086477-254d-49c8-83b8-9534c5fb88a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:21:41.211724 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.211560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps49g\" (UniqueName: \"kubernetes.io/projected/20086477-254d-49c8-83b8-9534c5fb88a5-kube-api-access-ps49g\") pod \"payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv\" (UID: \"20086477-254d-49c8-83b8-9534c5fb88a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:21:41.211724 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.211626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4300c43d-6031-48b8-a4b5-868df90b4700-plugins-config-volume\") pod \"payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd\" (UID: \"4300c43d-6031-48b8-a4b5-868df90b4700\") " pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:21:41.313003 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.312943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlc5n\" (UniqueName: \"kubernetes.io/projected/4300c43d-6031-48b8-a4b5-868df90b4700-kube-api-access-nlc5n\") pod \"payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd\" (UID: \"4300c43d-6031-48b8-a4b5-868df90b4700\") " pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:21:41.313003 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.313015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20086477-254d-49c8-83b8-9534c5fb88a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv\" (UID: \"20086477-254d-49c8-83b8-9534c5fb88a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:21:41.313248 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.313050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps49g\" (UniqueName: \"kubernetes.io/projected/20086477-254d-49c8-83b8-9534c5fb88a5-kube-api-access-ps49g\") pod \"payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv\" (UID: \"20086477-254d-49c8-83b8-9534c5fb88a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:21:41.313248 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.313094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4300c43d-6031-48b8-a4b5-868df90b4700-plugins-config-volume\") pod \"payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd\" (UID: \"4300c43d-6031-48b8-a4b5-868df90b4700\") " pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:21:41.314017 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.313994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4300c43d-6031-48b8-a4b5-868df90b4700-plugins-config-volume\") pod \"payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd\" (UID: \"4300c43d-6031-48b8-a4b5-868df90b4700\") " pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:21:41.314074 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.313994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20086477-254d-49c8-83b8-9534c5fb88a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv\" (UID: \"20086477-254d-49c8-83b8-9534c5fb88a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:21:41.321174 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.321149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps49g\" (UniqueName: \"kubernetes.io/projected/20086477-254d-49c8-83b8-9534c5fb88a5-kube-api-access-ps49g\") pod \"payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv\" (UID: \"20086477-254d-49c8-83b8-9534c5fb88a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:21:41.321305 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.321279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlc5n\" (UniqueName: \"kubernetes.io/projected/4300c43d-6031-48b8-a4b5-868df90b4700-kube-api-access-nlc5n\") pod \"payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd\" (UID: \"4300c43d-6031-48b8-a4b5-868df90b4700\") " pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:21:41.416797 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.416766    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:21:41.442256 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.442227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:21:41.697349 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.697326    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"]
Sep 09 03:21:41.699397 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:21:41.699363    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20086477_254d_49c8_83b8_9534c5fb88a5.slice/crio-e279d45251a4f2b926f197ba74bff44374b1be2c22ad8bfdb4e77dc4434f4d20 WatchSource:0}: Error finding container e279d45251a4f2b926f197ba74bff44374b1be2c22ad8bfdb4e77dc4434f4d20: Status 404 returned error can't find the container with id e279d45251a4f2b926f197ba74bff44374b1be2c22ad8bfdb4e77dc4434f4d20
Sep 09 03:21:41.728600 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.728551    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"]
Sep 09 03:21:41.731673 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:21:41.731641    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4300c43d_6031_48b8_a4b5_868df90b4700.slice/crio-624838807d951249a2da560d7466214686497a9933b50057eac07803e8eda0e4 WatchSource:0}: Error finding container 624838807d951249a2da560d7466214686497a9933b50057eac07803e8eda0e4: Status 404 returned error can't find the container with id 624838807d951249a2da560d7466214686497a9933b50057eac07803e8eda0e4
Sep 09 03:21:41.918699 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.918660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd" event={"ID":"4300c43d-6031-48b8-a4b5-868df90b4700","Type":"ContainerStarted","Data":"400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61"}
Sep 09 03:21:41.918915 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.918730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd" event={"ID":"4300c43d-6031-48b8-a4b5-868df90b4700","Type":"ContainerStarted","Data":"624838807d951249a2da560d7466214686497a9933b50057eac07803e8eda0e4"}
Sep 09 03:21:41.918915 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.918763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:21:41.920491 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.920469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv" event={"ID":"20086477-254d-49c8-83b8-9534c5fb88a5","Type":"ContainerStarted","Data":"cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13"}
Sep 09 03:21:41.920673 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.920513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv" event={"ID":"20086477-254d-49c8-83b8-9534c5fb88a5","Type":"ContainerStarted","Data":"e279d45251a4f2b926f197ba74bff44374b1be2c22ad8bfdb4e77dc4434f4d20"}
Sep 09 03:21:41.920673 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.920562    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:21:41.935604 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.935535    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd" podStartSLOduration=0.935520936 podStartE2EDuration="935.520936ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:41.934070202 +0000 UTC m=+945.460384940" watchObservedRunningTime="2026-09-09 03:21:41.935520936 +0000 UTC m=+945.461835675"
Sep 09 03:21:41.951390 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:41.951313    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv" podStartSLOduration=0.951299386 podStartE2EDuration="951.299386ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:41.949589931 +0000 UTC m=+945.475904666" watchObservedRunningTime="2026-09-09 03:21:41.951299386 +0000 UTC m=+945.477614125"
Sep 09 03:21:45.902505 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:45.902471    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:21:52.929785 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:52.929750    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:21:52.930318 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:21:52.930002    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:22:10.207973 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:10.207927    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:22:11.061637 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.061598    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"]
Sep 09 03:22:11.061855 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.061827    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd" podUID="4300c43d-6031-48b8-a4b5-868df90b4700" containerName="payload-processing" containerID="cri-o://400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61" gracePeriod=30
Sep 09 03:22:11.072053 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.072026    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"]
Sep 09 03:22:11.072275 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.072238    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv" podUID="20086477-254d-49c8-83b8-9534c5fb88a5" containerName="payload-pre-processing" containerID="cri-o://cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13" gracePeriod=30
Sep 09 03:22:11.357488 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.357461    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:22:11.360374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.360353    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:22:11.374601 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.374550    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlc5n\" (UniqueName: \"kubernetes.io/projected/4300c43d-6031-48b8-a4b5-868df90b4700-kube-api-access-nlc5n\") pod \"4300c43d-6031-48b8-a4b5-868df90b4700\" (UID: \"4300c43d-6031-48b8-a4b5-868df90b4700\") "
Sep 09 03:22:11.374748 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.374711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20086477-254d-49c8-83b8-9534c5fb88a5-plugins-config-volume\") pod \"20086477-254d-49c8-83b8-9534c5fb88a5\" (UID: \"20086477-254d-49c8-83b8-9534c5fb88a5\") "
Sep 09 03:22:11.375881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.374760    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4300c43d-6031-48b8-a4b5-868df90b4700-plugins-config-volume\") pod \"4300c43d-6031-48b8-a4b5-868df90b4700\" (UID: \"4300c43d-6031-48b8-a4b5-868df90b4700\") "
Sep 09 03:22:11.375881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.375802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20086477-254d-49c8-83b8-9534c5fb88a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20086477-254d-49c8-83b8-9534c5fb88a5" (UID: "20086477-254d-49c8-83b8-9534c5fb88a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:11.376025 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.375956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4300c43d-6031-48b8-a4b5-868df90b4700-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4300c43d-6031-48b8-a4b5-868df90b4700" (UID: "4300c43d-6031-48b8-a4b5-868df90b4700"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:11.377940 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.377916    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4300c43d-6031-48b8-a4b5-868df90b4700-kube-api-access-nlc5n" (OuterVolumeSpecName: "kube-api-access-nlc5n") pod "4300c43d-6031-48b8-a4b5-868df90b4700" (UID: "4300c43d-6031-48b8-a4b5-868df90b4700"). InnerVolumeSpecName "kube-api-access-nlc5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:11.476135 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.476096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps49g\" (UniqueName: \"kubernetes.io/projected/20086477-254d-49c8-83b8-9534c5fb88a5-kube-api-access-ps49g\") pod \"20086477-254d-49c8-83b8-9534c5fb88a5\" (UID: \"20086477-254d-49c8-83b8-9534c5fb88a5\") "
Sep 09 03:22:11.476360 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.476348    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlc5n\" (UniqueName: \"kubernetes.io/projected/4300c43d-6031-48b8-a4b5-868df90b4700-kube-api-access-nlc5n\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:11.476419 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.476364    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20086477-254d-49c8-83b8-9534c5fb88a5-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:11.476419 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.476372    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4300c43d-6031-48b8-a4b5-868df90b4700-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:11.478307 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.478282    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20086477-254d-49c8-83b8-9534c5fb88a5-kube-api-access-ps49g" (OuterVolumeSpecName: "kube-api-access-ps49g") pod "20086477-254d-49c8-83b8-9534c5fb88a5" (UID: "20086477-254d-49c8-83b8-9534c5fb88a5"). InnerVolumeSpecName "kube-api-access-ps49g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:11.577469 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:11.577440    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps49g\" (UniqueName: \"kubernetes.io/projected/20086477-254d-49c8-83b8-9534c5fb88a5-kube-api-access-ps49g\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:12.087668 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.087638    2580 generic.go:356] "Generic (PLEG): container finished" podID="4300c43d-6031-48b8-a4b5-868df90b4700" containerID="400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61" exitCode=0
Sep 09 03:22:12.087843 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.087683    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"
Sep 09 03:22:12.087843 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.087713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd" event={"ID":"4300c43d-6031-48b8-a4b5-868df90b4700","Type":"ContainerDied","Data":"400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61"}
Sep 09 03:22:12.087843 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.087764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd" event={"ID":"4300c43d-6031-48b8-a4b5-868df90b4700","Type":"ContainerDied","Data":"624838807d951249a2da560d7466214686497a9933b50057eac07803e8eda0e4"}
Sep 09 03:22:12.087843 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.087796    2580 scope.go:117] "RemoveContainer" containerID="400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61"
Sep 09 03:22:12.089262 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.089057    2580 generic.go:356] "Generic (PLEG): container finished" podID="20086477-254d-49c8-83b8-9534c5fb88a5" containerID="cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13" exitCode=0
Sep 09 03:22:12.089262 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.089145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv" event={"ID":"20086477-254d-49c8-83b8-9534c5fb88a5","Type":"ContainerDied","Data":"cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13"}
Sep 09 03:22:12.089262 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.089174    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"
Sep 09 03:22:12.089262 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.089187    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv" event={"ID":"20086477-254d-49c8-83b8-9534c5fb88a5","Type":"ContainerDied","Data":"e279d45251a4f2b926f197ba74bff44374b1be2c22ad8bfdb4e77dc4434f4d20"}
Sep 09 03:22:12.097980 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.097959    2580 scope.go:117] "RemoveContainer" containerID="400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61"
Sep 09 03:22:12.098338 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:22:12.098315    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61\": container with ID starting with 400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61 not found: ID does not exist" containerID="400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61"
Sep 09 03:22:12.098450 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.098349    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61"} err="failed to get container status \"400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61\": rpc error: code = NotFound desc = could not find container \"400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61\": container with ID starting with 400efbc01a60b85d1ece8ea420704f4d0bab6c3a0293c6e1ea1a7c53e6fc7e61 not found: ID does not exist"
Sep 09 03:22:12.098450 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.098371    2580 scope.go:117] "RemoveContainer" containerID="cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13"
Sep 09 03:22:12.107670 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.107651    2580 scope.go:117] "RemoveContainer" containerID="cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13"
Sep 09 03:22:12.107920 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:22:12.107901    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13\": container with ID starting with cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13 not found: ID does not exist" containerID="cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13"
Sep 09 03:22:12.107992 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.107930    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13"} err="failed to get container status \"cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13\": rpc error: code = NotFound desc = could not find container \"cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13\": container with ID starting with cdbd6c61cf583427132809972b7d3c810a0f80a22a3db6a1d461211b64261d13 not found: ID does not exist"
Sep 09 03:22:12.113999 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.113978    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"]
Sep 09 03:22:12.118093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.118073    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-189a872e-84588b5c4f-lw9qd"]
Sep 09 03:22:12.133041 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.133024    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"]
Sep 09 03:22:12.139509 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.139461    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-189a872e-6cbf448998-5c7jv"]
Sep 09 03:22:12.971151 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.971118    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20086477-254d-49c8-83b8-9534c5fb88a5" path="/var/lib/kubelet/pods/20086477-254d-49c8-83b8-9534c5fb88a5/volumes"
Sep 09 03:22:12.971528 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:12.971456    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4300c43d-6031-48b8-a4b5-868df90b4700" path="/var/lib/kubelet/pods/4300c43d-6031-48b8-a4b5-868df90b4700/volumes"
Sep 09 03:22:17.181872 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.181840    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"]
Sep 09 03:22:17.182263 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.182250    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20086477-254d-49c8-83b8-9534c5fb88a5" containerName="payload-pre-processing"
Sep 09 03:22:17.182309 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.182265    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="20086477-254d-49c8-83b8-9534c5fb88a5" containerName="payload-pre-processing"
Sep 09 03:22:17.182309 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.182282    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4300c43d-6031-48b8-a4b5-868df90b4700" containerName="payload-processing"
Sep 09 03:22:17.182309 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.182288    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4300c43d-6031-48b8-a4b5-868df90b4700" containerName="payload-processing"
Sep 09 03:22:17.182403 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.182354    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4300c43d-6031-48b8-a4b5-868df90b4700" containerName="payload-processing"
Sep 09 03:22:17.182403 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.182366    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="20086477-254d-49c8-83b8-9534c5fb88a5" containerName="payload-pre-processing"
Sep 09 03:22:17.187009 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.186989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:17.190004 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.189985    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8f2b65e3-dockercfg-nkqrz\""
Sep 09 03:22:17.190120 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.189989    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8f2b65e3\""
Sep 09 03:22:17.198370 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.198347    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"]
Sep 09 03:22:17.224643 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.224617    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"]
Sep 09 03:22:17.226136 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.226113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-plugins-config-volume\") pod \"payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p\" (UID: \"c7eac334-ce92-4ce0-9aa8-e3dc3c730023\") " pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:17.226209 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.226180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6br\" (UniqueName: \"kubernetes.io/projected/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-kube-api-access-nf6br\") pod \"payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p\" (UID: \"c7eac334-ce92-4ce0-9aa8-e3dc3c730023\") " pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:17.227903 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.227886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:17.235882 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.235856    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"]
Sep 09 03:22:17.326904 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.326854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6br\" (UniqueName: \"kubernetes.io/projected/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-kube-api-access-nf6br\") pod \"payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p\" (UID: \"c7eac334-ce92-4ce0-9aa8-e3dc3c730023\") " pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:17.327093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.326979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87\" (UID: \"d1be71a8-3403-47f1-9bbf-c40aa6dfa325\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:17.327093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.327044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-plugins-config-volume\") pod \"payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p\" (UID: \"c7eac334-ce92-4ce0-9aa8-e3dc3c730023\") " pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:17.327093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.327076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlsg\" (UniqueName: \"kubernetes.io/projected/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-kube-api-access-qmlsg\") pod \"payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87\" (UID: \"d1be71a8-3403-47f1-9bbf-c40aa6dfa325\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:17.327811 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.327792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-plugins-config-volume\") pod \"payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p\" (UID: \"c7eac334-ce92-4ce0-9aa8-e3dc3c730023\") " pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:17.335053 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.335032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6br\" (UniqueName: \"kubernetes.io/projected/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-kube-api-access-nf6br\") pod \"payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p\" (UID: \"c7eac334-ce92-4ce0-9aa8-e3dc3c730023\") " pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:17.427846 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.427802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87\" (UID: \"d1be71a8-3403-47f1-9bbf-c40aa6dfa325\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:17.428029 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.427901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlsg\" (UniqueName: \"kubernetes.io/projected/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-kube-api-access-qmlsg\") pod \"payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87\" (UID: \"d1be71a8-3403-47f1-9bbf-c40aa6dfa325\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:17.428744 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.428721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87\" (UID: \"d1be71a8-3403-47f1-9bbf-c40aa6dfa325\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:17.437229 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.437170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlsg\" (UniqueName: \"kubernetes.io/projected/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-kube-api-access-qmlsg\") pod \"payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87\" (UID: \"d1be71a8-3403-47f1-9bbf-c40aa6dfa325\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:17.498377 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.498342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:17.539507 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.539461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:17.775880 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.775857    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"]
Sep 09 03:22:17.777705 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:22:17.777678    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7eac334_ce92_4ce0_9aa8_e3dc3c730023.slice/crio-eb691866fab5a12b509e5d7d82963ccd31e86140b588d7a00a89e72464048f1f WatchSource:0}: Error finding container eb691866fab5a12b509e5d7d82963ccd31e86140b588d7a00a89e72464048f1f: Status 404 returned error can't find the container with id eb691866fab5a12b509e5d7d82963ccd31e86140b588d7a00a89e72464048f1f
Sep 09 03:22:17.822975 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:17.822889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"]
Sep 09 03:22:17.825765 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:22:17.825734    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1be71a8_3403_47f1_9bbf_c40aa6dfa325.slice/crio-4af3be7666713ac104edd868eb92fd43387bfb1898b06f4d0e5de3499628f1dd WatchSource:0}: Error finding container 4af3be7666713ac104edd868eb92fd43387bfb1898b06f4d0e5de3499628f1dd: Status 404 returned error can't find the container with id 4af3be7666713ac104edd868eb92fd43387bfb1898b06f4d0e5de3499628f1dd
Sep 09 03:22:18.122571 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:18.122531    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p" event={"ID":"c7eac334-ce92-4ce0-9aa8-e3dc3c730023","Type":"ContainerStarted","Data":"0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a"}
Sep 09 03:22:18.122772 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:18.122613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:18.122772 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:18.122639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p" event={"ID":"c7eac334-ce92-4ce0-9aa8-e3dc3c730023","Type":"ContainerStarted","Data":"eb691866fab5a12b509e5d7d82963ccd31e86140b588d7a00a89e72464048f1f"}
Sep 09 03:22:18.124229 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:18.124206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87" event={"ID":"d1be71a8-3403-47f1-9bbf-c40aa6dfa325","Type":"ContainerStarted","Data":"30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121"}
Sep 09 03:22:18.124354 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:18.124246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87" event={"ID":"d1be71a8-3403-47f1-9bbf-c40aa6dfa325","Type":"ContainerStarted","Data":"4af3be7666713ac104edd868eb92fd43387bfb1898b06f4d0e5de3499628f1dd"}
Sep 09 03:22:18.124396 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:18.124377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:18.143347 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:18.143293    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p" podStartSLOduration=1.143276849 podStartE2EDuration="1.143276849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:22:18.141038092 +0000 UTC m=+981.667352832" watchObservedRunningTime="2026-09-09 03:22:18.143276849 +0000 UTC m=+981.669591588"
Sep 09 03:22:18.161340 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:18.161281    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87" podStartSLOduration=1.161261337 podStartE2EDuration="1.161261337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:22:18.158879609 +0000 UTC m=+981.685194369" watchObservedRunningTime="2026-09-09 03:22:18.161261337 +0000 UTC m=+981.687576077"
Sep 09 03:22:27.512218 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:27.512178    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:22:29.131552 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:29.131522    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:29.131940 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:29.131611    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:38.924212 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:38.924173    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"]
Sep 09 03:22:38.924748 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:38.924435    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n" podUID="af89f475-897b-46aa-9e30-1d0051712b25" containerName="payload-processing" containerID="cri-o://dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88" gracePeriod=30
Sep 09 03:22:38.938528 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:38.938382    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"]
Sep 09 03:22:38.938666 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:38.938624    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz" podUID="7685bde2-1032-456a-87ab-cbcd4310b27d" containerName="payload-pre-processing" containerID="cri-o://950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17" gracePeriod=30
Sep 09 03:22:39.218308 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.218282    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:22:39.221245 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.221224    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:22:39.226526 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.226501    2580 generic.go:356] "Generic (PLEG): container finished" podID="af89f475-897b-46aa-9e30-1d0051712b25" containerID="dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88" exitCode=0
Sep 09 03:22:39.226664 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.226536    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"
Sep 09 03:22:39.226664 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.226590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n" event={"ID":"af89f475-897b-46aa-9e30-1d0051712b25","Type":"ContainerDied","Data":"dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88"}
Sep 09 03:22:39.226664 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.226634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n" event={"ID":"af89f475-897b-46aa-9e30-1d0051712b25","Type":"ContainerDied","Data":"cb7c34807f4c09b4348ab30276f91ea576ac4fd76660598ec8e7de14132a7bd0"}
Sep 09 03:22:39.226664 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.226656    2580 scope.go:117] "RemoveContainer" containerID="dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88"
Sep 09 03:22:39.228185 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.228135    2580 generic.go:356] "Generic (PLEG): container finished" podID="7685bde2-1032-456a-87ab-cbcd4310b27d" containerID="950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17" exitCode=0
Sep 09 03:22:39.228185 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.228171    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"
Sep 09 03:22:39.228348 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.228185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz" event={"ID":"7685bde2-1032-456a-87ab-cbcd4310b27d","Type":"ContainerDied","Data":"950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17"}
Sep 09 03:22:39.228348 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.228256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz" event={"ID":"7685bde2-1032-456a-87ab-cbcd4310b27d","Type":"ContainerDied","Data":"ba64141ff3d7aad76af8d091b4477bebf03cc9f5227198683c836a0bd5af4bd8"}
Sep 09 03:22:39.235217 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.235203    2580 scope.go:117] "RemoveContainer" containerID="dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88"
Sep 09 03:22:39.235471 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:22:39.235450    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88\": container with ID starting with dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88 not found: ID does not exist" containerID="dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88"
Sep 09 03:22:39.235555 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.235477    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88"} err="failed to get container status \"dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88\": rpc error: code = NotFound desc = could not find container \"dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88\": container with ID starting with dfbf4ba79c40f26a035db39166283fb7317fcc6a1152d6114d8f05072890ab88 not found: ID does not exist"
Sep 09 03:22:39.235555 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.235495    2580 scope.go:117] "RemoveContainer" containerID="950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17"
Sep 09 03:22:39.243079 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.243063    2580 scope.go:117] "RemoveContainer" containerID="950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17"
Sep 09 03:22:39.243349 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:22:39.243333    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17\": container with ID starting with 950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17 not found: ID does not exist" containerID="950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17"
Sep 09 03:22:39.243411 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.243361    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17"} err="failed to get container status \"950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17\": rpc error: code = NotFound desc = could not find container \"950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17\": container with ID starting with 950baf86a9b78e27b54ae32db8e0993b19127a655d19668beeef518fcf6c4b17 not found: ID does not exist"
Sep 09 03:22:39.327432 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.327399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7685bde2-1032-456a-87ab-cbcd4310b27d-plugins-config-volume\") pod \"7685bde2-1032-456a-87ab-cbcd4310b27d\" (UID: \"7685bde2-1032-456a-87ab-cbcd4310b27d\") "
Sep 09 03:22:39.327637 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.327494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af89f475-897b-46aa-9e30-1d0051712b25-plugins-config-volume\") pod \"af89f475-897b-46aa-9e30-1d0051712b25\" (UID: \"af89f475-897b-46aa-9e30-1d0051712b25\") "
Sep 09 03:22:39.327637 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.327526    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sdd9\" (UniqueName: \"kubernetes.io/projected/7685bde2-1032-456a-87ab-cbcd4310b27d-kube-api-access-9sdd9\") pod \"7685bde2-1032-456a-87ab-cbcd4310b27d\" (UID: \"7685bde2-1032-456a-87ab-cbcd4310b27d\") "
Sep 09 03:22:39.327637 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.327621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m6w8\" (UniqueName: \"kubernetes.io/projected/af89f475-897b-46aa-9e30-1d0051712b25-kube-api-access-4m6w8\") pod \"af89f475-897b-46aa-9e30-1d0051712b25\" (UID: \"af89f475-897b-46aa-9e30-1d0051712b25\") "
Sep 09 03:22:39.327870 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.327844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7685bde2-1032-456a-87ab-cbcd4310b27d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7685bde2-1032-456a-87ab-cbcd4310b27d" (UID: "7685bde2-1032-456a-87ab-cbcd4310b27d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:39.327915 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.327880    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af89f475-897b-46aa-9e30-1d0051712b25-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af89f475-897b-46aa-9e30-1d0051712b25" (UID: "af89f475-897b-46aa-9e30-1d0051712b25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:39.329807 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.329787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7685bde2-1032-456a-87ab-cbcd4310b27d-kube-api-access-9sdd9" (OuterVolumeSpecName: "kube-api-access-9sdd9") pod "7685bde2-1032-456a-87ab-cbcd4310b27d" (UID: "7685bde2-1032-456a-87ab-cbcd4310b27d"). InnerVolumeSpecName "kube-api-access-9sdd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:39.329857 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.329809    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af89f475-897b-46aa-9e30-1d0051712b25-kube-api-access-4m6w8" (OuterVolumeSpecName: "kube-api-access-4m6w8") pod "af89f475-897b-46aa-9e30-1d0051712b25" (UID: "af89f475-897b-46aa-9e30-1d0051712b25"). InnerVolumeSpecName "kube-api-access-4m6w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:39.428256 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.428224    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m6w8\" (UniqueName: \"kubernetes.io/projected/af89f475-897b-46aa-9e30-1d0051712b25-kube-api-access-4m6w8\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:39.428256 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.428253    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7685bde2-1032-456a-87ab-cbcd4310b27d-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:39.428256 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.428263    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af89f475-897b-46aa-9e30-1d0051712b25-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:39.428491 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.428273    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sdd9\" (UniqueName: \"kubernetes.io/projected/7685bde2-1032-456a-87ab-cbcd4310b27d-kube-api-access-9sdd9\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:39.552265 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.552234    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"]
Sep 09 03:22:39.556360 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.556333    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-be414f-68cbc79fcd-5dqxz"]
Sep 09 03:22:39.570531 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.570504    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"]
Sep 09 03:22:39.573659 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:39.573627    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-be414f-5c67fd4dc7-kqs5n"]
Sep 09 03:22:40.228975 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.228898    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"]
Sep 09 03:22:40.229375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.229327    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af89f475-897b-46aa-9e30-1d0051712b25" containerName="payload-processing"
Sep 09 03:22:40.229375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.229340    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af89f475-897b-46aa-9e30-1d0051712b25" containerName="payload-processing"
Sep 09 03:22:40.229375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.229351    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7685bde2-1032-456a-87ab-cbcd4310b27d" containerName="payload-pre-processing"
Sep 09 03:22:40.229375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.229356    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7685bde2-1032-456a-87ab-cbcd4310b27d" containerName="payload-pre-processing"
Sep 09 03:22:40.229520 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.229410    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af89f475-897b-46aa-9e30-1d0051712b25" containerName="payload-processing"
Sep 09 03:22:40.229520 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.229420    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7685bde2-1032-456a-87ab-cbcd4310b27d" containerName="payload-pre-processing"
Sep 09 03:22:40.239460 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.239435    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"
Sep 09 03:22:40.242566 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.242543    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:22:40.243063 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.243045    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:22:40.248347 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.248327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"]
Sep 09 03:22:40.340342 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.340300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90eec3d1-c96c-4e7f-a919-6ba281369936-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n\" (UID: \"90eec3d1-c96c-4e7f-a919-6ba281369936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"
Sep 09 03:22:40.441543 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.441508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90eec3d1-c96c-4e7f-a919-6ba281369936-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n\" (UID: \"90eec3d1-c96c-4e7f-a919-6ba281369936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"
Sep 09 03:22:40.442224 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.442203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90eec3d1-c96c-4e7f-a919-6ba281369936-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n\" (UID: \"90eec3d1-c96c-4e7f-a919-6ba281369936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"
Sep 09 03:22:40.551171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.551081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"
Sep 09 03:22:40.830174 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.830093    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"]
Sep 09 03:22:40.833800 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:22:40.833773    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90eec3d1_c96c_4e7f_a919_6ba281369936.slice/crio-b982395472c95fc466f4732306911f8f3eb35962a38d26f89b46e7c063963d14 WatchSource:0}: Error finding container b982395472c95fc466f4732306911f8f3eb35962a38d26f89b46e7c063963d14: Status 404 returned error can't find the container with id b982395472c95fc466f4732306911f8f3eb35962a38d26f89b46e7c063963d14
Sep 09 03:22:40.971517 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.971484    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7685bde2-1032-456a-87ab-cbcd4310b27d" path="/var/lib/kubelet/pods/7685bde2-1032-456a-87ab-cbcd4310b27d/volumes"
Sep 09 03:22:40.971899 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:40.971884    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af89f475-897b-46aa-9e30-1d0051712b25" path="/var/lib/kubelet/pods/af89f475-897b-46aa-9e30-1d0051712b25/volumes"
Sep 09 03:22:41.247948 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:41.247912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n" event={"ID":"90eec3d1-c96c-4e7f-a919-6ba281369936","Type":"ContainerStarted","Data":"b982395472c95fc466f4732306911f8f3eb35962a38d26f89b46e7c063963d14"}
Sep 09 03:22:44.266949 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:44.266914    2580 generic.go:356] "Generic (PLEG): container finished" podID="90eec3d1-c96c-4e7f-a919-6ba281369936" containerID="266ff27ebb2e373be55f4c56061c99edac1fd9975f48165ec2635ec4f02d9d6b" exitCode=0
Sep 09 03:22:44.267344 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:44.266950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n" event={"ID":"90eec3d1-c96c-4e7f-a919-6ba281369936","Type":"ContainerDied","Data":"266ff27ebb2e373be55f4c56061c99edac1fd9975f48165ec2635ec4f02d9d6b"}
Sep 09 03:22:45.400217 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:45.400192    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"
Sep 09 03:22:45.488924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:45.488893    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90eec3d1-c96c-4e7f-a919-6ba281369936-maas-api-service-ca\") pod \"90eec3d1-c96c-4e7f-a919-6ba281369936\" (UID: \"90eec3d1-c96c-4e7f-a919-6ba281369936\") "
Sep 09 03:22:45.489267 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:45.489242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90eec3d1-c96c-4e7f-a919-6ba281369936-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "90eec3d1-c96c-4e7f-a919-6ba281369936" (UID: "90eec3d1-c96c-4e7f-a919-6ba281369936"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:45.589922 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:45.589845    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90eec3d1-c96c-4e7f-a919-6ba281369936-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:46.283750 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:46.283721    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"
Sep 09 03:22:46.283922 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:46.283759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n" event={"ID":"90eec3d1-c96c-4e7f-a919-6ba281369936","Type":"ContainerDied","Data":"b982395472c95fc466f4732306911f8f3eb35962a38d26f89b46e7c063963d14"}
Sep 09 03:22:46.283922 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:46.283825    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b982395472c95fc466f4732306911f8f3eb35962a38d26f89b46e7c063963d14"
Sep 09 03:22:47.245703 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.245672    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"]
Sep 09 03:22:47.246137 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.245892    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p" podUID="c7eac334-ce92-4ce0-9aa8-e3dc3c730023" containerName="payload-processing" containerID="cri-o://0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a" gracePeriod=30
Sep 09 03:22:47.254901 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.254856    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"]
Sep 09 03:22:47.255083 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.255059    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87" podUID="d1be71a8-3403-47f1-9bbf-c40aa6dfa325" containerName="payload-pre-processing" containerID="cri-o://30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121" gracePeriod=30
Sep 09 03:22:47.460187 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.460149    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"]
Sep 09 03:22:47.462224 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.462201    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8f2b65e3-t6s4n"]
Sep 09 03:22:47.534165 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.534140    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:47.537341 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.537323    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:47.708760 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.708711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-plugins-config-volume\") pod \"c7eac334-ce92-4ce0-9aa8-e3dc3c730023\" (UID: \"c7eac334-ce92-4ce0-9aa8-e3dc3c730023\") "
Sep 09 03:22:47.708760 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.708750    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmlsg\" (UniqueName: \"kubernetes.io/projected/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-kube-api-access-qmlsg\") pod \"d1be71a8-3403-47f1-9bbf-c40aa6dfa325\" (UID: \"d1be71a8-3403-47f1-9bbf-c40aa6dfa325\") "
Sep 09 03:22:47.709031 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.708835    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-plugins-config-volume\") pod \"d1be71a8-3403-47f1-9bbf-c40aa6dfa325\" (UID: \"d1be71a8-3403-47f1-9bbf-c40aa6dfa325\") "
Sep 09 03:22:47.709031 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.708891    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf6br\" (UniqueName: \"kubernetes.io/projected/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-kube-api-access-nf6br\") pod \"c7eac334-ce92-4ce0-9aa8-e3dc3c730023\" (UID: \"c7eac334-ce92-4ce0-9aa8-e3dc3c730023\") "
Sep 09 03:22:47.709559 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.709508    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1be71a8-3403-47f1-9bbf-c40aa6dfa325" (UID: "d1be71a8-3403-47f1-9bbf-c40aa6dfa325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:47.709717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.709634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7eac334-ce92-4ce0-9aa8-e3dc3c730023" (UID: "c7eac334-ce92-4ce0-9aa8-e3dc3c730023"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:47.711597 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.711552    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-kube-api-access-nf6br" (OuterVolumeSpecName: "kube-api-access-nf6br") pod "c7eac334-ce92-4ce0-9aa8-e3dc3c730023" (UID: "c7eac334-ce92-4ce0-9aa8-e3dc3c730023"). InnerVolumeSpecName "kube-api-access-nf6br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:47.711823 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.711798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-kube-api-access-qmlsg" (OuterVolumeSpecName: "kube-api-access-qmlsg") pod "d1be71a8-3403-47f1-9bbf-c40aa6dfa325" (UID: "d1be71a8-3403-47f1-9bbf-c40aa6dfa325"). InnerVolumeSpecName "kube-api-access-qmlsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:22:47.810619 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.810501    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:47.810619 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.810535    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlsg\" (UniqueName: \"kubernetes.io/projected/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-kube-api-access-qmlsg\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:47.810619 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.810551    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1be71a8-3403-47f1-9bbf-c40aa6dfa325-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:47.810619 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:47.810564    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf6br\" (UniqueName: \"kubernetes.io/projected/c7eac334-ce92-4ce0-9aa8-e3dc3c730023-kube-api-access-nf6br\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:22:48.295100 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.295065    2580 generic.go:356] "Generic (PLEG): container finished" podID="c7eac334-ce92-4ce0-9aa8-e3dc3c730023" containerID="0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a" exitCode=0
Sep 09 03:22:48.295553 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.295111    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p" event={"ID":"c7eac334-ce92-4ce0-9aa8-e3dc3c730023","Type":"ContainerDied","Data":"0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a"}
Sep 09 03:22:48.295553 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.295134    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"
Sep 09 03:22:48.295553 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.295162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p" event={"ID":"c7eac334-ce92-4ce0-9aa8-e3dc3c730023","Type":"ContainerDied","Data":"eb691866fab5a12b509e5d7d82963ccd31e86140b588d7a00a89e72464048f1f"}
Sep 09 03:22:48.295553 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.295192    2580 scope.go:117] "RemoveContainer" containerID="0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a"
Sep 09 03:22:48.296865 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.296845    2580 generic.go:356] "Generic (PLEG): container finished" podID="d1be71a8-3403-47f1-9bbf-c40aa6dfa325" containerID="30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121" exitCode=0
Sep 09 03:22:48.297005 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.296977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87" event={"ID":"d1be71a8-3403-47f1-9bbf-c40aa6dfa325","Type":"ContainerDied","Data":"30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121"}
Sep 09 03:22:48.297005 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.296997    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"
Sep 09 03:22:48.297137 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.297036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87" event={"ID":"d1be71a8-3403-47f1-9bbf-c40aa6dfa325","Type":"ContainerDied","Data":"4af3be7666713ac104edd868eb92fd43387bfb1898b06f4d0e5de3499628f1dd"}
Sep 09 03:22:48.305046 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.305028    2580 scope.go:117] "RemoveContainer" containerID="0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a"
Sep 09 03:22:48.305352 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:22:48.305329    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a\": container with ID starting with 0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a not found: ID does not exist" containerID="0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a"
Sep 09 03:22:48.305448 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.305364    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a"} err="failed to get container status \"0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a\": rpc error: code = NotFound desc = could not find container \"0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a\": container with ID starting with 0ea9373f399270a4dace7b053e5753a75eeaa475a687e51fd0f7d8dcd9513a5a not found: ID does not exist"
Sep 09 03:22:48.305448 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.305391    2580 scope.go:117] "RemoveContainer" containerID="30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121"
Sep 09 03:22:48.314360 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.314331    2580 scope.go:117] "RemoveContainer" containerID="30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121"
Sep 09 03:22:48.314741 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:22:48.314714    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121\": container with ID starting with 30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121 not found: ID does not exist" containerID="30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121"
Sep 09 03:22:48.314802 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.314749    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121"} err="failed to get container status \"30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121\": rpc error: code = NotFound desc = could not find container \"30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121\": container with ID starting with 30280a96ebe7b2b6a58821e777c0a42ff6efec4cc94aedd8bf095513ff8be121 not found: ID does not exist"
Sep 09 03:22:48.335772 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:22:48.335748    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1be71a8_3403_47f1_9bbf_c40aa6dfa325.slice\": RecentStats: unable to find data in memory cache]"
Sep 09 03:22:48.353348 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.353317    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"]
Sep 09 03:22:48.363373 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.363345    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8f2b65e3-6c69f4f466-hps87"]
Sep 09 03:22:48.403186 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.403153    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"]
Sep 09 03:22:48.412118 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.412088    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8f2b65e3-86cf684497-tlw2p"]
Sep 09 03:22:48.971620 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.971557    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90eec3d1-c96c-4e7f-a919-6ba281369936" path="/var/lib/kubelet/pods/90eec3d1-c96c-4e7f-a919-6ba281369936/volumes"
Sep 09 03:22:48.971969 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.971955    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7eac334-ce92-4ce0-9aa8-e3dc3c730023" path="/var/lib/kubelet/pods/c7eac334-ce92-4ce0-9aa8-e3dc3c730023/volumes"
Sep 09 03:22:48.972265 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:48.972253    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1be71a8-3403-47f1-9bbf-c40aa6dfa325" path="/var/lib/kubelet/pods/d1be71a8-3403-47f1-9bbf-c40aa6dfa325/volumes"
Sep 09 03:22:53.308247 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:22:53.308204    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:23:11.193377 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:11.193343    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:23:19.807080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:19.807043    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:23:27.524656 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.524625    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"]
Sep 09 03:23:27.525048 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.525035    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90eec3d1-c96c-4e7f-a919-6ba281369936" containerName="revoke-keys"
Sep 09 03:23:27.525093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.525050    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="90eec3d1-c96c-4e7f-a919-6ba281369936" containerName="revoke-keys"
Sep 09 03:23:27.525093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.525062    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7eac334-ce92-4ce0-9aa8-e3dc3c730023" containerName="payload-processing"
Sep 09 03:23:27.525093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.525067    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7eac334-ce92-4ce0-9aa8-e3dc3c730023" containerName="payload-processing"
Sep 09 03:23:27.525093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.525084    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1be71a8-3403-47f1-9bbf-c40aa6dfa325" containerName="payload-pre-processing"
Sep 09 03:23:27.525093 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.525089    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1be71a8-3403-47f1-9bbf-c40aa6dfa325" containerName="payload-pre-processing"
Sep 09 03:23:27.525253 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.525153    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7eac334-ce92-4ce0-9aa8-e3dc3c730023" containerName="payload-processing"
Sep 09 03:23:27.525253 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.525160    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="90eec3d1-c96c-4e7f-a919-6ba281369936" containerName="revoke-keys"
Sep 09 03:23:27.525253 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.525168    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1be71a8-3403-47f1-9bbf-c40aa6dfa325" containerName="payload-pre-processing"
Sep 09 03:23:27.528225 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.528211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:23:27.530698 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.530675    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-4ce4c1-dockercfg-rwbbt\""
Sep 09 03:23:27.530969 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.530953    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-4ce4c1\""
Sep 09 03:23:27.544862 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.544834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"]
Sep 09 03:23:27.577121 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.577090    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"]
Sep 09 03:23:27.580535 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.580517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:23:27.603715 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.603693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"]
Sep 09 03:23:27.639549 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.639517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdbs\" (UniqueName: \"kubernetes.io/projected/3e3b2427-af6b-4d25-a87e-3e05db4fae12-kube-api-access-5sdbs\") pod \"payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg\" (UID: \"3e3b2427-af6b-4d25-a87e-3e05db4fae12\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:23:27.639741 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.639643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcd2s\" (UniqueName: \"kubernetes.io/projected/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-kube-api-access-lcd2s\") pod \"payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn\" (UID: \"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:23:27.639741 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.639676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3b2427-af6b-4d25-a87e-3e05db4fae12-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg\" (UID: \"3e3b2427-af6b-4d25-a87e-3e05db4fae12\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:23:27.639741 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.639704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn\" (UID: \"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:23:27.740498 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.740464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcd2s\" (UniqueName: \"kubernetes.io/projected/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-kube-api-access-lcd2s\") pod \"payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn\" (UID: \"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:23:27.740498 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.740502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3b2427-af6b-4d25-a87e-3e05db4fae12-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg\" (UID: \"3e3b2427-af6b-4d25-a87e-3e05db4fae12\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:23:27.740754 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.740533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn\" (UID: \"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:23:27.740754 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.740564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdbs\" (UniqueName: \"kubernetes.io/projected/3e3b2427-af6b-4d25-a87e-3e05db4fae12-kube-api-access-5sdbs\") pod \"payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg\" (UID: \"3e3b2427-af6b-4d25-a87e-3e05db4fae12\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:23:27.741287 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.741265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3b2427-af6b-4d25-a87e-3e05db4fae12-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg\" (UID: \"3e3b2427-af6b-4d25-a87e-3e05db4fae12\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:23:27.741339 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.741295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn\" (UID: \"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:23:27.750921 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.750886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdbs\" (UniqueName: \"kubernetes.io/projected/3e3b2427-af6b-4d25-a87e-3e05db4fae12-kube-api-access-5sdbs\") pod \"payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg\" (UID: \"3e3b2427-af6b-4d25-a87e-3e05db4fae12\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:23:27.751037 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.751018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcd2s\" (UniqueName: \"kubernetes.io/projected/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-kube-api-access-lcd2s\") pod \"payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn\" (UID: \"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:23:27.837808 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.837734    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:23:27.890118 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:27.889814    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:23:28.111000 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:28.110968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"]
Sep 09 03:23:28.114393 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:23:28.114368    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3b2427_af6b_4d25_a87e_3e05db4fae12.slice/crio-1af81d3d9ed15f12c0d78d02d89f5b3c4999684a17e85baebec3b0c5a3dd71c4 WatchSource:0}: Error finding container 1af81d3d9ed15f12c0d78d02d89f5b3c4999684a17e85baebec3b0c5a3dd71c4: Status 404 returned error can't find the container with id 1af81d3d9ed15f12c0d78d02d89f5b3c4999684a17e85baebec3b0c5a3dd71c4
Sep 09 03:23:28.180210 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:28.180188    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"]
Sep 09 03:23:28.182154 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:23:28.182133    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0ca02d_833c_4aa4_a56e_4b8ba8e016f8.slice/crio-3d1e079ca21d8165ff1ecc2f28af8a30d2f431b5116f2f9a2f8dc49ca1e834e0 WatchSource:0}: Error finding container 3d1e079ca21d8165ff1ecc2f28af8a30d2f431b5116f2f9a2f8dc49ca1e834e0: Status 404 returned error can't find the container with id 3d1e079ca21d8165ff1ecc2f28af8a30d2f431b5116f2f9a2f8dc49ca1e834e0
Sep 09 03:23:28.503715 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:28.503607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg" event={"ID":"3e3b2427-af6b-4d25-a87e-3e05db4fae12","Type":"ContainerStarted","Data":"5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8"}
Sep 09 03:23:28.503893 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:28.503770    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:23:28.503893 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:28.503804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg" event={"ID":"3e3b2427-af6b-4d25-a87e-3e05db4fae12","Type":"ContainerStarted","Data":"1af81d3d9ed15f12c0d78d02d89f5b3c4999684a17e85baebec3b0c5a3dd71c4"}
Sep 09 03:23:28.505210 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:28.505184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn" event={"ID":"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8","Type":"ContainerStarted","Data":"e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0"}
Sep 09 03:23:28.505345 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:28.505242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn" event={"ID":"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8","Type":"ContainerStarted","Data":"3d1e079ca21d8165ff1ecc2f28af8a30d2f431b5116f2f9a2f8dc49ca1e834e0"}
Sep 09 03:23:28.505345 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:28.505268    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:23:28.529348 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:28.529293    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg" podStartSLOduration=1.5292737189999999 podStartE2EDuration="1.529273719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:23:28.529202063 +0000 UTC m=+1052.055516802" watchObservedRunningTime="2026-09-09 03:23:28.529273719 +0000 UTC m=+1052.055588459"
Sep 09 03:23:33.313625 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:33.313554    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn" podStartSLOduration=6.313538485 podStartE2EDuration="6.313538485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:23:28.575130533 +0000 UTC m=+1052.101445272" watchObservedRunningTime="2026-09-09 03:23:33.313538485 +0000 UTC m=+1056.839853224"
Sep 09 03:23:33.314515 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:33.314498    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:23:39.512397 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:39.512368    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:23:39.512863 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:39.512474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:23:53.795521 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:23:53.795479    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:24:08.269429 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:08.269390    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:24:26.102808 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:26.102770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:24:37.334999 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.334954    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"]
Sep 09 03:24:37.335624 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.335223    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg" podUID="3e3b2427-af6b-4d25-a87e-3e05db4fae12" containerName="payload-processing" containerID="cri-o://5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8" gracePeriod=30
Sep 09 03:24:37.340357 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.340327    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"]
Sep 09 03:24:37.340868 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.340820    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn" podUID="fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8" containerName="payload-pre-processing" containerID="cri-o://e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0" gracePeriod=30
Sep 09 03:24:37.626510 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.626483    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:24:37.629740 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.629712    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:24:37.679755 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.679711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sdbs\" (UniqueName: \"kubernetes.io/projected/3e3b2427-af6b-4d25-a87e-3e05db4fae12-kube-api-access-5sdbs\") pod \"3e3b2427-af6b-4d25-a87e-3e05db4fae12\" (UID: \"3e3b2427-af6b-4d25-a87e-3e05db4fae12\") "
Sep 09 03:24:37.679955 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.679777    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcd2s\" (UniqueName: \"kubernetes.io/projected/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-kube-api-access-lcd2s\") pod \"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8\" (UID: \"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8\") "
Sep 09 03:24:37.679955 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.679804    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-plugins-config-volume\") pod \"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8\" (UID: \"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8\") "
Sep 09 03:24:37.679955 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.679858    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3b2427-af6b-4d25-a87e-3e05db4fae12-plugins-config-volume\") pod \"3e3b2427-af6b-4d25-a87e-3e05db4fae12\" (UID: \"3e3b2427-af6b-4d25-a87e-3e05db4fae12\") "
Sep 09 03:24:37.680220 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.680190    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8" (UID: "fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:24:37.680337 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.680280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3b2427-af6b-4d25-a87e-3e05db4fae12-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e3b2427-af6b-4d25-a87e-3e05db4fae12" (UID: "3e3b2427-af6b-4d25-a87e-3e05db4fae12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:24:37.682161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.682129    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-kube-api-access-lcd2s" (OuterVolumeSpecName: "kube-api-access-lcd2s") pod "fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8" (UID: "fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8"). InnerVolumeSpecName "kube-api-access-lcd2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:24:37.682161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.682143    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3b2427-af6b-4d25-a87e-3e05db4fae12-kube-api-access-5sdbs" (OuterVolumeSpecName: "kube-api-access-5sdbs") pod "3e3b2427-af6b-4d25-a87e-3e05db4fae12" (UID: "3e3b2427-af6b-4d25-a87e-3e05db4fae12"). InnerVolumeSpecName "kube-api-access-5sdbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:24:37.781479 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.781446    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sdbs\" (UniqueName: \"kubernetes.io/projected/3e3b2427-af6b-4d25-a87e-3e05db4fae12-kube-api-access-5sdbs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:24:37.781479 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.781474    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcd2s\" (UniqueName: \"kubernetes.io/projected/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-kube-api-access-lcd2s\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:24:37.781479 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.781484    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:24:37.781747 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.781494    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3b2427-af6b-4d25-a87e-3e05db4fae12-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:24:37.850833 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.850796    2580 generic.go:356] "Generic (PLEG): container finished" podID="3e3b2427-af6b-4d25-a87e-3e05db4fae12" containerID="5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8" exitCode=0
Sep 09 03:24:37.851030 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.850849    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"
Sep 09 03:24:37.851030 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.850880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg" event={"ID":"3e3b2427-af6b-4d25-a87e-3e05db4fae12","Type":"ContainerDied","Data":"5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8"}
Sep 09 03:24:37.851030 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.850936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg" event={"ID":"3e3b2427-af6b-4d25-a87e-3e05db4fae12","Type":"ContainerDied","Data":"1af81d3d9ed15f12c0d78d02d89f5b3c4999684a17e85baebec3b0c5a3dd71c4"}
Sep 09 03:24:37.851030 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.850966    2580 scope.go:117] "RemoveContainer" containerID="5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8"
Sep 09 03:24:37.852299 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.852202    2580 generic.go:356] "Generic (PLEG): container finished" podID="fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8" containerID="e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0" exitCode=0
Sep 09 03:24:37.852299 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.852244    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"
Sep 09 03:24:37.852299 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.852258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn" event={"ID":"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8","Type":"ContainerDied","Data":"e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0"}
Sep 09 03:24:37.852299 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.852299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn" event={"ID":"fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8","Type":"ContainerDied","Data":"3d1e079ca21d8165ff1ecc2f28af8a30d2f431b5116f2f9a2f8dc49ca1e834e0"}
Sep 09 03:24:37.860039 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.860013    2580 scope.go:117] "RemoveContainer" containerID="5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8"
Sep 09 03:24:37.860353 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:24:37.860333    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8\": container with ID starting with 5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8 not found: ID does not exist" containerID="5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8"
Sep 09 03:24:37.860450 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.860359    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8"} err="failed to get container status \"5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8\": rpc error: code = NotFound desc = could not find container \"5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8\": container with ID starting with 5ca47ae7bddee734238773f1182ab55426ef130354480f254161c1058fb9c9f8 not found: ID does not exist"
Sep 09 03:24:37.860450 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.860377    2580 scope.go:117] "RemoveContainer" containerID="e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0"
Sep 09 03:24:37.868386 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.868364    2580 scope.go:117] "RemoveContainer" containerID="e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0"
Sep 09 03:24:37.868651 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:24:37.868635    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0\": container with ID starting with e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0 not found: ID does not exist" containerID="e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0"
Sep 09 03:24:37.868716 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.868657    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0"} err="failed to get container status \"e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0\": rpc error: code = NotFound desc = could not find container \"e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0\": container with ID starting with e4d448cf383a5d6eea896b345bf9892c1b86c89c24718c7bb8850c246d5718f0 not found: ID does not exist"
Sep 09 03:24:37.875083 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.875055    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"]
Sep 09 03:24:37.876991 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.876969    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ce4c1-85d6585c85-drqjg"]
Sep 09 03:24:37.885199 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.885178    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"]
Sep 09 03:24:37.891501 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:37.891479    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ce4c1-8689c45dfd-x22mn"]
Sep 09 03:24:38.960654 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.960617    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"]
Sep 09 03:24:38.961206 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.961186    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e3b2427-af6b-4d25-a87e-3e05db4fae12" containerName="payload-processing"
Sep 09 03:24:38.961301 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.961210    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e3b2427-af6b-4d25-a87e-3e05db4fae12" containerName="payload-processing"
Sep 09 03:24:38.961301 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.961227    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8" containerName="payload-pre-processing"
Sep 09 03:24:38.961301 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.961237    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8" containerName="payload-pre-processing"
Sep 09 03:24:38.961451 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.961320    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8" containerName="payload-pre-processing"
Sep 09 03:24:38.961451 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.961334    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e3b2427-af6b-4d25-a87e-3e05db4fae12" containerName="payload-processing"
Sep 09 03:24:38.965686 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.965664    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:24:38.970261 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.970237    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-57aecdff-dockercfg-sln48\""
Sep 09 03:24:38.970412 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.970237    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-57aecdff\""
Sep 09 03:24:38.972008 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.971978    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e3b2427-af6b-4d25-a87e-3e05db4fae12" path="/var/lib/kubelet/pods/3e3b2427-af6b-4d25-a87e-3e05db4fae12/volumes"
Sep 09 03:24:38.972407 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.972391    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8" path="/var/lib/kubelet/pods/fe0ca02d-833c-4aa4-a56e-4b8ba8e016f8/volumes"
Sep 09 03:24:38.980268 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.980243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"]
Sep 09 03:24:38.993781 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.993751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f566f27-09cf-4664-a680-3d06cdab4082-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz\" (UID: \"2f566f27-09cf-4664-a680-3d06cdab4082\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:24:38.993944 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.993801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5lnf\" (UniqueName: \"kubernetes.io/projected/2f566f27-09cf-4664-a680-3d06cdab4082-kube-api-access-k5lnf\") pod \"payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz\" (UID: \"2f566f27-09cf-4664-a680-3d06cdab4082\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:24:38.996863 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:38.996836    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"]
Sep 09 03:24:39.000411 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.000391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:24:39.012170 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.012140    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"]
Sep 09 03:24:39.094870 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.094835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5lnf\" (UniqueName: \"kubernetes.io/projected/2f566f27-09cf-4664-a680-3d06cdab4082-kube-api-access-k5lnf\") pod \"payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz\" (UID: \"2f566f27-09cf-4664-a680-3d06cdab4082\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:24:39.095044 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.094903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ct48\" (UniqueName: \"kubernetes.io/projected/ebdfc281-143e-421c-bdd7-6aec391f95fa-kube-api-access-4ct48\") pod \"payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7\" (UID: \"ebdfc281-143e-421c-bdd7-6aec391f95fa\") " pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:24:39.095044 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.094949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebdfc281-143e-421c-bdd7-6aec391f95fa-plugins-config-volume\") pod \"payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7\" (UID: \"ebdfc281-143e-421c-bdd7-6aec391f95fa\") " pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:24:39.095044 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.094970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f566f27-09cf-4664-a680-3d06cdab4082-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz\" (UID: \"2f566f27-09cf-4664-a680-3d06cdab4082\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:24:39.095626 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.095607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f566f27-09cf-4664-a680-3d06cdab4082-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz\" (UID: \"2f566f27-09cf-4664-a680-3d06cdab4082\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:24:39.102642 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.102619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5lnf\" (UniqueName: \"kubernetes.io/projected/2f566f27-09cf-4664-a680-3d06cdab4082-kube-api-access-k5lnf\") pod \"payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz\" (UID: \"2f566f27-09cf-4664-a680-3d06cdab4082\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:24:39.196012 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.195974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ct48\" (UniqueName: \"kubernetes.io/projected/ebdfc281-143e-421c-bdd7-6aec391f95fa-kube-api-access-4ct48\") pod \"payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7\" (UID: \"ebdfc281-143e-421c-bdd7-6aec391f95fa\") " pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:24:39.196196 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.196031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebdfc281-143e-421c-bdd7-6aec391f95fa-plugins-config-volume\") pod \"payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7\" (UID: \"ebdfc281-143e-421c-bdd7-6aec391f95fa\") " pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:24:39.196735 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.196717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebdfc281-143e-421c-bdd7-6aec391f95fa-plugins-config-volume\") pod \"payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7\" (UID: \"ebdfc281-143e-421c-bdd7-6aec391f95fa\") " pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:24:39.203448 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.203426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ct48\" (UniqueName: \"kubernetes.io/projected/ebdfc281-143e-421c-bdd7-6aec391f95fa-kube-api-access-4ct48\") pod \"payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7\" (UID: \"ebdfc281-143e-421c-bdd7-6aec391f95fa\") " pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:24:39.276321 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.276249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:24:39.312257 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.312224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:24:39.550352 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.550329    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"]
Sep 09 03:24:39.552673 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:24:39.552644    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f566f27_09cf_4664_a680_3d06cdab4082.slice/crio-a216c906778d78e63e1fc69ff0ef137aca40399985ae61202899cd54e10726ff WatchSource:0}: Error finding container a216c906778d78e63e1fc69ff0ef137aca40399985ae61202899cd54e10726ff: Status 404 returned error can't find the container with id a216c906778d78e63e1fc69ff0ef137aca40399985ae61202899cd54e10726ff
Sep 09 03:24:39.592696 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.592661    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"]
Sep 09 03:24:39.604636 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:24:39.604609    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebdfc281_143e_421c_bdd7_6aec391f95fa.slice/crio-77c06222ff86e969554e74ba602704d684b9f436cf65c19bc27314d83bb8f98f WatchSource:0}: Error finding container 77c06222ff86e969554e74ba602704d684b9f436cf65c19bc27314d83bb8f98f: Status 404 returned error can't find the container with id 77c06222ff86e969554e74ba602704d684b9f436cf65c19bc27314d83bb8f98f
Sep 09 03:24:39.866216 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.866178    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7" event={"ID":"ebdfc281-143e-421c-bdd7-6aec391f95fa","Type":"ContainerStarted","Data":"4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10"}
Sep 09 03:24:39.866440 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.866244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7" event={"ID":"ebdfc281-143e-421c-bdd7-6aec391f95fa","Type":"ContainerStarted","Data":"77c06222ff86e969554e74ba602704d684b9f436cf65c19bc27314d83bb8f98f"}
Sep 09 03:24:39.866440 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.866306    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:24:39.867881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.867858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz" event={"ID":"2f566f27-09cf-4664-a680-3d06cdab4082","Type":"ContainerStarted","Data":"165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b"}
Sep 09 03:24:39.867994 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.867897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:24:39.867994 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.867913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz" event={"ID":"2f566f27-09cf-4664-a680-3d06cdab4082","Type":"ContainerStarted","Data":"a216c906778d78e63e1fc69ff0ef137aca40399985ae61202899cd54e10726ff"}
Sep 09 03:24:39.881602 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.881539    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7" podStartSLOduration=1.881523853 podStartE2EDuration="1.881523853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:24:39.880312106 +0000 UTC m=+1123.406626847" watchObservedRunningTime="2026-09-09 03:24:39.881523853 +0000 UTC m=+1123.407838650"
Sep 09 03:24:39.896885 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:39.896817    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz" podStartSLOduration=1.896800461 podStartE2EDuration="1.896800461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:24:39.894795494 +0000 UTC m=+1123.421110233" watchObservedRunningTime="2026-09-09 03:24:39.896800461 +0000 UTC m=+1123.423115199"
Sep 09 03:24:42.802156 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:42.802116    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:24:50.875746 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:50.875715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:24:50.876205 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:50.875798    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:24:58.496345 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:24:58.496315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:25:19.971617 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:19.971510    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"]
Sep 09 03:25:19.972150 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:19.971801    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7" podUID="ebdfc281-143e-421c-bdd7-6aec391f95fa" containerName="payload-processing" containerID="cri-o://4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10" gracePeriod=30
Sep 09 03:25:19.977438 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:19.977408    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"]
Sep 09 03:25:19.977693 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:19.977647    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz" podUID="2f566f27-09cf-4664-a680-3d06cdab4082" containerName="payload-pre-processing" containerID="cri-o://165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b" gracePeriod=30
Sep 09 03:25:20.254172 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.254151    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:25:20.257010 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.256990    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:25:20.273077 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.272773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ct48\" (UniqueName: \"kubernetes.io/projected/ebdfc281-143e-421c-bdd7-6aec391f95fa-kube-api-access-4ct48\") pod \"ebdfc281-143e-421c-bdd7-6aec391f95fa\" (UID: \"ebdfc281-143e-421c-bdd7-6aec391f95fa\") "
Sep 09 03:25:20.273077 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.272826    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebdfc281-143e-421c-bdd7-6aec391f95fa-plugins-config-volume\") pod \"ebdfc281-143e-421c-bdd7-6aec391f95fa\" (UID: \"ebdfc281-143e-421c-bdd7-6aec391f95fa\") "
Sep 09 03:25:20.273311 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.273260    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebdfc281-143e-421c-bdd7-6aec391f95fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ebdfc281-143e-421c-bdd7-6aec391f95fa" (UID: "ebdfc281-143e-421c-bdd7-6aec391f95fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:20.275459 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.275436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdfc281-143e-421c-bdd7-6aec391f95fa-kube-api-access-4ct48" (OuterVolumeSpecName: "kube-api-access-4ct48") pod "ebdfc281-143e-421c-bdd7-6aec391f95fa" (UID: "ebdfc281-143e-421c-bdd7-6aec391f95fa"). InnerVolumeSpecName "kube-api-access-4ct48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:20.374102 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.374065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f566f27-09cf-4664-a680-3d06cdab4082-plugins-config-volume\") pod \"2f566f27-09cf-4664-a680-3d06cdab4082\" (UID: \"2f566f27-09cf-4664-a680-3d06cdab4082\") "
Sep 09 03:25:20.374304 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.374141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5lnf\" (UniqueName: \"kubernetes.io/projected/2f566f27-09cf-4664-a680-3d06cdab4082-kube-api-access-k5lnf\") pod \"2f566f27-09cf-4664-a680-3d06cdab4082\" (UID: \"2f566f27-09cf-4664-a680-3d06cdab4082\") "
Sep 09 03:25:20.374304 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.374294    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ct48\" (UniqueName: \"kubernetes.io/projected/ebdfc281-143e-421c-bdd7-6aec391f95fa-kube-api-access-4ct48\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:20.374304 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.374304    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebdfc281-143e-421c-bdd7-6aec391f95fa-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:20.374474 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.374380    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f566f27-09cf-4664-a680-3d06cdab4082-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f566f27-09cf-4664-a680-3d06cdab4082" (UID: "2f566f27-09cf-4664-a680-3d06cdab4082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:20.376358 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.376334    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f566f27-09cf-4664-a680-3d06cdab4082-kube-api-access-k5lnf" (OuterVolumeSpecName: "kube-api-access-k5lnf") pod "2f566f27-09cf-4664-a680-3d06cdab4082" (UID: "2f566f27-09cf-4664-a680-3d06cdab4082"). InnerVolumeSpecName "kube-api-access-k5lnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:20.475468 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.475436    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f566f27-09cf-4664-a680-3d06cdab4082-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:20.475468 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:20.475462    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5lnf\" (UniqueName: \"kubernetes.io/projected/2f566f27-09cf-4664-a680-3d06cdab4082-kube-api-access-k5lnf\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:21.066202 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.066166    2580 generic.go:356] "Generic (PLEG): container finished" podID="ebdfc281-143e-421c-bdd7-6aec391f95fa" containerID="4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10" exitCode=0
Sep 09 03:25:21.066717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.066211    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"
Sep 09 03:25:21.066717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.066247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7" event={"ID":"ebdfc281-143e-421c-bdd7-6aec391f95fa","Type":"ContainerDied","Data":"4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10"}
Sep 09 03:25:21.066717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.066304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7" event={"ID":"ebdfc281-143e-421c-bdd7-6aec391f95fa","Type":"ContainerDied","Data":"77c06222ff86e969554e74ba602704d684b9f436cf65c19bc27314d83bb8f98f"}
Sep 09 03:25:21.066717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.066328    2580 scope.go:117] "RemoveContainer" containerID="4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10"
Sep 09 03:25:21.067873 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.067721    2580 generic.go:356] "Generic (PLEG): container finished" podID="2f566f27-09cf-4664-a680-3d06cdab4082" containerID="165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b" exitCode=0
Sep 09 03:25:21.067873 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.067770    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"
Sep 09 03:25:21.067873 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.067810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz" event={"ID":"2f566f27-09cf-4664-a680-3d06cdab4082","Type":"ContainerDied","Data":"165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b"}
Sep 09 03:25:21.067873 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.067872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz" event={"ID":"2f566f27-09cf-4664-a680-3d06cdab4082","Type":"ContainerDied","Data":"a216c906778d78e63e1fc69ff0ef137aca40399985ae61202899cd54e10726ff"}
Sep 09 03:25:21.075338 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.075321    2580 scope.go:117] "RemoveContainer" containerID="4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10"
Sep 09 03:25:21.075642 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:25:21.075619    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10\": container with ID starting with 4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10 not found: ID does not exist" containerID="4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10"
Sep 09 03:25:21.075724 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.075655    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10"} err="failed to get container status \"4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10\": rpc error: code = NotFound desc = could not find container \"4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10\": container with ID starting with 4dd37614d0bc4b4b7b07b76f94c3e821446d5330839187ac7db07adf13b44a10 not found: ID does not exist"
Sep 09 03:25:21.075724 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.075676    2580 scope.go:117] "RemoveContainer" containerID="165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b"
Sep 09 03:25:21.083991 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.083968    2580 scope.go:117] "RemoveContainer" containerID="165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b"
Sep 09 03:25:21.084298 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:25:21.084280    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b\": container with ID starting with 165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b not found: ID does not exist" containerID="165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b"
Sep 09 03:25:21.084379 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.084309    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b"} err="failed to get container status \"165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b\": rpc error: code = NotFound desc = could not find container \"165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b\": container with ID starting with 165b54ddf24d51f772f802c002f83a122385f8f3ed6833f02ef019034c49aa8b not found: ID does not exist"
Sep 09 03:25:21.087463 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.087393    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"]
Sep 09 03:25:21.089365 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.089344    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57aecdff-68599ddf57-cn9t7"]
Sep 09 03:25:21.100080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.100058    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"]
Sep 09 03:25:21.104327 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.104303    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57aecdff-94ff79658-m8jgz"]
Sep 09 03:25:21.399747 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:21.399710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:25:22.971490 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:22.971457    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f566f27-09cf-4664-a680-3d06cdab4082" path="/var/lib/kubelet/pods/2f566f27-09cf-4664-a680-3d06cdab4082/volumes"
Sep 09 03:25:22.971892 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:22.971840    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebdfc281-143e-421c-bdd7-6aec391f95fa" path="/var/lib/kubelet/pods/ebdfc281-143e-421c-bdd7-6aec391f95fa/volumes"
Sep 09 03:25:27.843528 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:27.843493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"]
Sep 09 03:25:27.844082 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:27.843721    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl" podUID="95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2" containerName="payload-processing" containerID="cri-o://40daf5e06a08f68189bf254e53c9b7ec44ba502f864ec012d27b9b0d656f5674" gracePeriod=30
Sep 09 03:25:27.852303 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:27.852268    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"]
Sep 09 03:25:27.852804 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:27.852551    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429" podUID="1c2e3179-a7aa-4f52-a37d-74985be48cf0" containerName="payload-pre-processing" containerID="cri-o://952bb87458e67bff8312b58b1f4d2406565836d8b4cd2855c4b6be23f7fb7fe9" gracePeriod=30
Sep 09 03:25:28.102859 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.102827    2580 generic.go:356] "Generic (PLEG): container finished" podID="1c2e3179-a7aa-4f52-a37d-74985be48cf0" containerID="952bb87458e67bff8312b58b1f4d2406565836d8b4cd2855c4b6be23f7fb7fe9" exitCode=0
Sep 09 03:25:28.103015 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.102907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429" event={"ID":"1c2e3179-a7aa-4f52-a37d-74985be48cf0","Type":"ContainerDied","Data":"952bb87458e67bff8312b58b1f4d2406565836d8b4cd2855c4b6be23f7fb7fe9"}
Sep 09 03:25:28.104490 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.104467    2580 generic.go:356] "Generic (PLEG): container finished" podID="95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2" containerID="40daf5e06a08f68189bf254e53c9b7ec44ba502f864ec012d27b9b0d656f5674" exitCode=0
Sep 09 03:25:28.104697 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.104510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl" event={"ID":"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2","Type":"ContainerDied","Data":"40daf5e06a08f68189bf254e53c9b7ec44ba502f864ec012d27b9b0d656f5674"}
Sep 09 03:25:28.120632 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.120606    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:25:28.132374 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.132356    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:25:28.249216 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.249187    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c2e3179-a7aa-4f52-a37d-74985be48cf0-plugins-config-volume\") pod \"1c2e3179-a7aa-4f52-a37d-74985be48cf0\" (UID: \"1c2e3179-a7aa-4f52-a37d-74985be48cf0\") "
Sep 09 03:25:28.249411 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.249223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwbmg\" (UniqueName: \"kubernetes.io/projected/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-kube-api-access-dwbmg\") pod \"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2\" (UID: \"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2\") "
Sep 09 03:25:28.249411 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.249287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltxzb\" (UniqueName: \"kubernetes.io/projected/1c2e3179-a7aa-4f52-a37d-74985be48cf0-kube-api-access-ltxzb\") pod \"1c2e3179-a7aa-4f52-a37d-74985be48cf0\" (UID: \"1c2e3179-a7aa-4f52-a37d-74985be48cf0\") "
Sep 09 03:25:28.249411 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.249325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-plugins-config-volume\") pod \"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2\" (UID: \"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2\") "
Sep 09 03:25:28.249744 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.249712    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2e3179-a7aa-4f52-a37d-74985be48cf0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c2e3179-a7aa-4f52-a37d-74985be48cf0" (UID: "1c2e3179-a7aa-4f52-a37d-74985be48cf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:28.249744 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.249726    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2" (UID: "95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:28.251552 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.251532    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2e3179-a7aa-4f52-a37d-74985be48cf0-kube-api-access-ltxzb" (OuterVolumeSpecName: "kube-api-access-ltxzb") pod "1c2e3179-a7aa-4f52-a37d-74985be48cf0" (UID: "1c2e3179-a7aa-4f52-a37d-74985be48cf0"). InnerVolumeSpecName "kube-api-access-ltxzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:28.251679 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.251608    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-kube-api-access-dwbmg" (OuterVolumeSpecName: "kube-api-access-dwbmg") pod "95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2" (UID: "95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2"). InnerVolumeSpecName "kube-api-access-dwbmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:28.350505 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.350452    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltxzb\" (UniqueName: \"kubernetes.io/projected/1c2e3179-a7aa-4f52-a37d-74985be48cf0-kube-api-access-ltxzb\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:28.350505 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.350499    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:28.350505 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.350509    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c2e3179-a7aa-4f52-a37d-74985be48cf0-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:28.350505 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:28.350518    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwbmg\" (UniqueName: \"kubernetes.io/projected/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2-kube-api-access-dwbmg\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:29.111435 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:29.111358    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"
Sep 09 03:25:29.111435 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:29.111375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429" event={"ID":"1c2e3179-a7aa-4f52-a37d-74985be48cf0","Type":"ContainerDied","Data":"453d9156d003e4408daf0edcfac4a885c689c0b055e430c21d2e49a39e0d45ec"}
Sep 09 03:25:29.111435 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:29.111433    2580 scope.go:117] "RemoveContainer" containerID="952bb87458e67bff8312b58b1f4d2406565836d8b4cd2855c4b6be23f7fb7fe9"
Sep 09 03:25:29.113028 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:29.113001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl" event={"ID":"95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2","Type":"ContainerDied","Data":"85068d4627362c720a88b8e79591cc6949d876ea8d8971f3e691c309282ab1f0"}
Sep 09 03:25:29.113028 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:29.113026    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"
Sep 09 03:25:29.120227 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:29.120212    2580 scope.go:117] "RemoveContainer" containerID="40daf5e06a08f68189bf254e53c9b7ec44ba502f864ec012d27b9b0d656f5674"
Sep 09 03:25:29.130918 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:29.130897    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"]
Sep 09 03:25:29.134058 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:29.134037    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b27dde-b78fb5549-ltthl"]
Sep 09 03:25:29.148021 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:29.147995    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"]
Sep 09 03:25:29.151935 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:29.151914    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b27dde-75cbc9c858-h2429"]
Sep 09 03:25:30.971427 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:30.971392    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2e3179-a7aa-4f52-a37d-74985be48cf0" path="/var/lib/kubelet/pods/1c2e3179-a7aa-4f52-a37d-74985be48cf0/volumes"
Sep 09 03:25:30.971820 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:30.971769    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2" path="/var/lib/kubelet/pods/95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2/volumes"
Sep 09 03:25:34.714340 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:34.714305    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:25:36.139110 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139073    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"]
Sep 09 03:25:36.139505 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139491    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebdfc281-143e-421c-bdd7-6aec391f95fa" containerName="payload-processing"
Sep 09 03:25:36.139545 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139507    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebdfc281-143e-421c-bdd7-6aec391f95fa" containerName="payload-processing"
Sep 09 03:25:36.139545 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139525    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2" containerName="payload-processing"
Sep 09 03:25:36.139545 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139531    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2" containerName="payload-processing"
Sep 09 03:25:36.139545 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139540    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2e3179-a7aa-4f52-a37d-74985be48cf0" containerName="payload-pre-processing"
Sep 09 03:25:36.139714 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139549    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2e3179-a7aa-4f52-a37d-74985be48cf0" containerName="payload-pre-processing"
Sep 09 03:25:36.139714 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139559    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f566f27-09cf-4664-a680-3d06cdab4082" containerName="payload-pre-processing"
Sep 09 03:25:36.139714 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139565    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f566f27-09cf-4664-a680-3d06cdab4082" containerName="payload-pre-processing"
Sep 09 03:25:36.139714 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139643    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f566f27-09cf-4664-a680-3d06cdab4082" containerName="payload-pre-processing"
Sep 09 03:25:36.139714 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139653    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c0bebc-ab07-4bc9-a0a1-e3d35dd068d2" containerName="payload-processing"
Sep 09 03:25:36.139714 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139659    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2e3179-a7aa-4f52-a37d-74985be48cf0" containerName="payload-pre-processing"
Sep 09 03:25:36.139714 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.139664    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebdfc281-143e-421c-bdd7-6aec391f95fa" containerName="payload-processing"
Sep 09 03:25:36.144219 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.144199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"
Sep 09 03:25:36.146849 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.146825    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:25:36.147036 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.147019    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:25:36.157514 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.157495    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"]
Sep 09 03:25:36.222769 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.222738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae138c7-3ffe-4cff-8476-4ca5674cb2ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl\" (UID: \"eae138c7-3ffe-4cff-8476-4ca5674cb2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"
Sep 09 03:25:36.324035 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.323998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae138c7-3ffe-4cff-8476-4ca5674cb2ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl\" (UID: \"eae138c7-3ffe-4cff-8476-4ca5674cb2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"
Sep 09 03:25:36.324724 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.324698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae138c7-3ffe-4cff-8476-4ca5674cb2ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl\" (UID: \"eae138c7-3ffe-4cff-8476-4ca5674cb2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"
Sep 09 03:25:36.454237 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.454152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"
Sep 09 03:25:36.732214 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:36.732132    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"]
Sep 09 03:25:36.739364 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:25:36.739331    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae138c7_3ffe_4cff_8476_4ca5674cb2ba.slice/crio-c86e64cfd241027dcf6eb76ff47c8c94c38cbcbac8bfabf95bc70d28ee89e9fb WatchSource:0}: Error finding container c86e64cfd241027dcf6eb76ff47c8c94c38cbcbac8bfabf95bc70d28ee89e9fb: Status 404 returned error can't find the container with id c86e64cfd241027dcf6eb76ff47c8c94c38cbcbac8bfabf95bc70d28ee89e9fb
Sep 09 03:25:37.154808 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:37.154772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl" event={"ID":"eae138c7-3ffe-4cff-8476-4ca5674cb2ba","Type":"ContainerStarted","Data":"be253c297535308e359fc43f2e2abaab5f6559f9e0d07aeb18843a81a060f46a"}
Sep 09 03:25:37.155255 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:37.154829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl" event={"ID":"eae138c7-3ffe-4cff-8476-4ca5674cb2ba","Type":"ContainerStarted","Data":"c86e64cfd241027dcf6eb76ff47c8c94c38cbcbac8bfabf95bc70d28ee89e9fb"}
Sep 09 03:25:37.178314 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:37.178264    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl" podStartSLOduration=1.178249081 podStartE2EDuration="1.178249081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:25:37.176330363 +0000 UTC m=+1180.702645103" watchObservedRunningTime="2026-09-09 03:25:37.178249081 +0000 UTC m=+1180.704563820"
Sep 09 03:25:38.161262 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:38.161223    2580 generic.go:356] "Generic (PLEG): container finished" podID="eae138c7-3ffe-4cff-8476-4ca5674cb2ba" containerID="be253c297535308e359fc43f2e2abaab5f6559f9e0d07aeb18843a81a060f46a" exitCode=0
Sep 09 03:25:38.161726 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:38.161294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl" event={"ID":"eae138c7-3ffe-4cff-8476-4ca5674cb2ba","Type":"ContainerDied","Data":"be253c297535308e359fc43f2e2abaab5f6559f9e0d07aeb18843a81a060f46a"}
Sep 09 03:25:39.295176 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:39.295143    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"
Sep 09 03:25:39.453316 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:39.453228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae138c7-3ffe-4cff-8476-4ca5674cb2ba-maas-api-service-ca\") pod \"eae138c7-3ffe-4cff-8476-4ca5674cb2ba\" (UID: \"eae138c7-3ffe-4cff-8476-4ca5674cb2ba\") "
Sep 09 03:25:39.453615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:39.453593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae138c7-3ffe-4cff-8476-4ca5674cb2ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eae138c7-3ffe-4cff-8476-4ca5674cb2ba" (UID: "eae138c7-3ffe-4cff-8476-4ca5674cb2ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:39.554661 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:39.554620    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae138c7-3ffe-4cff-8476-4ca5674cb2ba-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:40.171761 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:40.171732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"
Sep 09 03:25:40.171941 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:40.171764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl" event={"ID":"eae138c7-3ffe-4cff-8476-4ca5674cb2ba","Type":"ContainerDied","Data":"c86e64cfd241027dcf6eb76ff47c8c94c38cbcbac8bfabf95bc70d28ee89e9fb"}
Sep 09 03:25:40.171941 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:40.171809    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c86e64cfd241027dcf6eb76ff47c8c94c38cbcbac8bfabf95bc70d28ee89e9fb"
Sep 09 03:25:45.096127 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.096091    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"]
Sep 09 03:25:45.096507 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.096444    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae138c7-3ffe-4cff-8476-4ca5674cb2ba" containerName="revoke-keys"
Sep 09 03:25:45.096507 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.096455    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae138c7-3ffe-4cff-8476-4ca5674cb2ba" containerName="revoke-keys"
Sep 09 03:25:45.096606 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.096529    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae138c7-3ffe-4cff-8476-4ca5674cb2ba" containerName="revoke-keys"
Sep 09 03:25:45.099632 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.099614    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:25:45.101900 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.101879    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-207ebd\""
Sep 09 03:25:45.110313 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.110285    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"]
Sep 09 03:25:45.132510 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.132480    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"]
Sep 09 03:25:45.135899 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.135883    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:25:45.138161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.138141    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-207ebd-dockercfg-rf5gz\""
Sep 09 03:25:45.148167 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.148144    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"]
Sep 09 03:25:45.208518 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.208470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860317db-ee8c-4f93-863d-7ac3ead55cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn\" (UID: \"860317db-ee8c-4f93-863d-7ac3ead55cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:25:45.208751 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.208538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdprl\" (UniqueName: \"kubernetes.io/projected/860317db-ee8c-4f93-863d-7ac3ead55cc1-kube-api-access-hdprl\") pod \"payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn\" (UID: \"860317db-ee8c-4f93-863d-7ac3ead55cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:25:45.310045 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.310010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63947e97-2e0e-49df-8805-b3ec7a626f04-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p\" (UID: \"63947e97-2e0e-49df-8805-b3ec7a626f04\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:25:45.310222 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.310062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjb7b\" (UniqueName: \"kubernetes.io/projected/63947e97-2e0e-49df-8805-b3ec7a626f04-kube-api-access-cjb7b\") pod \"payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p\" (UID: \"63947e97-2e0e-49df-8805-b3ec7a626f04\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:25:45.310222 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.310127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860317db-ee8c-4f93-863d-7ac3ead55cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn\" (UID: \"860317db-ee8c-4f93-863d-7ac3ead55cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:25:45.310222 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.310160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdprl\" (UniqueName: \"kubernetes.io/projected/860317db-ee8c-4f93-863d-7ac3ead55cc1-kube-api-access-hdprl\") pod \"payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn\" (UID: \"860317db-ee8c-4f93-863d-7ac3ead55cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:25:45.310798 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.310780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860317db-ee8c-4f93-863d-7ac3ead55cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn\" (UID: \"860317db-ee8c-4f93-863d-7ac3ead55cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:25:45.319473 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.319441    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdprl\" (UniqueName: \"kubernetes.io/projected/860317db-ee8c-4f93-863d-7ac3ead55cc1-kube-api-access-hdprl\") pod \"payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn\" (UID: \"860317db-ee8c-4f93-863d-7ac3ead55cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:25:45.410748 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.410718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:25:45.410974 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.410952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63947e97-2e0e-49df-8805-b3ec7a626f04-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p\" (UID: \"63947e97-2e0e-49df-8805-b3ec7a626f04\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:25:45.411029 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.410998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjb7b\" (UniqueName: \"kubernetes.io/projected/63947e97-2e0e-49df-8805-b3ec7a626f04-kube-api-access-cjb7b\") pod \"payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p\" (UID: \"63947e97-2e0e-49df-8805-b3ec7a626f04\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:25:45.411636 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.411620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63947e97-2e0e-49df-8805-b3ec7a626f04-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p\" (UID: \"63947e97-2e0e-49df-8805-b3ec7a626f04\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:25:45.419814 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.419793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjb7b\" (UniqueName: \"kubernetes.io/projected/63947e97-2e0e-49df-8805-b3ec7a626f04-kube-api-access-cjb7b\") pod \"payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p\" (UID: \"63947e97-2e0e-49df-8805-b3ec7a626f04\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:25:45.445749 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.445712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:25:45.686375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.686346    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"]
Sep 09 03:25:45.688520 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:25:45.688494    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860317db_ee8c_4f93_863d_7ac3ead55cc1.slice/crio-ff68f06e3e1fde4abbb38983c52d1a2381b5c735eef2915da0bf8b6e699b04f6 WatchSource:0}: Error finding container ff68f06e3e1fde4abbb38983c52d1a2381b5c735eef2915da0bf8b6e699b04f6: Status 404 returned error can't find the container with id ff68f06e3e1fde4abbb38983c52d1a2381b5c735eef2915da0bf8b6e699b04f6
Sep 09 03:25:45.729537 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:45.729491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"]
Sep 09 03:25:45.735700 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:25:45.735672    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63947e97_2e0e_49df_8805_b3ec7a626f04.slice/crio-b6938b8a6656e42e10f1a2313968fdf86705bad6e55fb47f968c9ae73c199655 WatchSource:0}: Error finding container b6938b8a6656e42e10f1a2313968fdf86705bad6e55fb47f968c9ae73c199655: Status 404 returned error can't find the container with id b6938b8a6656e42e10f1a2313968fdf86705bad6e55fb47f968c9ae73c199655
Sep 09 03:25:46.205112 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.205063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn" event={"ID":"860317db-ee8c-4f93-863d-7ac3ead55cc1","Type":"ContainerStarted","Data":"2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae"}
Sep 09 03:25:46.205625 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.205564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn" event={"ID":"860317db-ee8c-4f93-863d-7ac3ead55cc1","Type":"ContainerStarted","Data":"ff68f06e3e1fde4abbb38983c52d1a2381b5c735eef2915da0bf8b6e699b04f6"}
Sep 09 03:25:46.205789 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.205765    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:25:46.208139 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.207875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p" event={"ID":"63947e97-2e0e-49df-8805-b3ec7a626f04","Type":"ContainerStarted","Data":"13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560"}
Sep 09 03:25:46.208139 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.207974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p" event={"ID":"63947e97-2e0e-49df-8805-b3ec7a626f04","Type":"ContainerStarted","Data":"b6938b8a6656e42e10f1a2313968fdf86705bad6e55fb47f968c9ae73c199655"}
Sep 09 03:25:46.208410 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.208378    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:25:46.225814 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.225761    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn" podStartSLOduration=1.225746928 podStartE2EDuration="1.225746928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:25:46.22309432 +0000 UTC m=+1189.749409071" watchObservedRunningTime="2026-09-09 03:25:46.225746928 +0000 UTC m=+1189.752061667"
Sep 09 03:25:46.251019 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.250967    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p" podStartSLOduration=1.250952412 podStartE2EDuration="1.250952412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:25:46.248093279 +0000 UTC m=+1189.774408017" watchObservedRunningTime="2026-09-09 03:25:46.250952412 +0000 UTC m=+1189.777267185"
Sep 09 03:25:46.402777 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.402737    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"]
Sep 09 03:25:46.403016 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.402990    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw" podUID="bd22869f-85a0-40e6-b9d8-55b21ae1da6b" containerName="payload-processing" containerID="cri-o://e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8" gracePeriod=30
Sep 09 03:25:46.409751 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.409206    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"]
Sep 09 03:25:46.409751 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.409399    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj" podUID="f5173d2d-efe2-42b8-b25c-b2f0b5c46857" containerName="payload-pre-processing" containerID="cri-o://a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1" gracePeriod=30
Sep 09 03:25:46.576510 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.576344    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"]
Sep 09 03:25:46.580603 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.579978    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70784b-rwlwl"]
Sep 09 03:25:46.678327 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.678242    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:25:46.694522 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.694499    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:25:46.825750 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.825718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbfx6\" (UniqueName: \"kubernetes.io/projected/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-kube-api-access-cbfx6\") pod \"bd22869f-85a0-40e6-b9d8-55b21ae1da6b\" (UID: \"bd22869f-85a0-40e6-b9d8-55b21ae1da6b\") "
Sep 09 03:25:46.825958 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.825804    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-plugins-config-volume\") pod \"f5173d2d-efe2-42b8-b25c-b2f0b5c46857\" (UID: \"f5173d2d-efe2-42b8-b25c-b2f0b5c46857\") "
Sep 09 03:25:46.825958 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.825852    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-plugins-config-volume\") pod \"bd22869f-85a0-40e6-b9d8-55b21ae1da6b\" (UID: \"bd22869f-85a0-40e6-b9d8-55b21ae1da6b\") "
Sep 09 03:25:46.825958 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.825903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4844c\" (UniqueName: \"kubernetes.io/projected/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-kube-api-access-4844c\") pod \"f5173d2d-efe2-42b8-b25c-b2f0b5c46857\" (UID: \"f5173d2d-efe2-42b8-b25c-b2f0b5c46857\") "
Sep 09 03:25:46.826283 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.826249    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd22869f-85a0-40e6-b9d8-55b21ae1da6b" (UID: "bd22869f-85a0-40e6-b9d8-55b21ae1da6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:46.826423 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.826248    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5173d2d-efe2-42b8-b25c-b2f0b5c46857" (UID: "f5173d2d-efe2-42b8-b25c-b2f0b5c46857"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:46.828072 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.828049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-kube-api-access-cbfx6" (OuterVolumeSpecName: "kube-api-access-cbfx6") pod "bd22869f-85a0-40e6-b9d8-55b21ae1da6b" (UID: "bd22869f-85a0-40e6-b9d8-55b21ae1da6b"). InnerVolumeSpecName "kube-api-access-cbfx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:46.828228 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.828209    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-kube-api-access-4844c" (OuterVolumeSpecName: "kube-api-access-4844c") pod "f5173d2d-efe2-42b8-b25c-b2f0b5c46857" (UID: "f5173d2d-efe2-42b8-b25c-b2f0b5c46857"). InnerVolumeSpecName "kube-api-access-4844c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:46.927041 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.926945    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbfx6\" (UniqueName: \"kubernetes.io/projected/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-kube-api-access-cbfx6\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:46.927041 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.926975    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:46.927041 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.926985    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd22869f-85a0-40e6-b9d8-55b21ae1da6b-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:46.927041 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.926996    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4844c\" (UniqueName: \"kubernetes.io/projected/f5173d2d-efe2-42b8-b25c-b2f0b5c46857-kube-api-access-4844c\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:25:46.971725 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:46.971692    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eae138c7-3ffe-4cff-8476-4ca5674cb2ba" path="/var/lib/kubelet/pods/eae138c7-3ffe-4cff-8476-4ca5674cb2ba/volumes"
Sep 09 03:25:47.215928 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.215827    2580 generic.go:356] "Generic (PLEG): container finished" podID="f5173d2d-efe2-42b8-b25c-b2f0b5c46857" containerID="a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1" exitCode=0
Sep 09 03:25:47.215928 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.215872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj" event={"ID":"f5173d2d-efe2-42b8-b25c-b2f0b5c46857","Type":"ContainerDied","Data":"a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1"}
Sep 09 03:25:47.215928 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.215903    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"
Sep 09 03:25:47.216452 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.215939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj" event={"ID":"f5173d2d-efe2-42b8-b25c-b2f0b5c46857","Type":"ContainerDied","Data":"43d7ebbb7e2cdf981a7448136a88aaf0be71adaa866314214dc37939369ea779"}
Sep 09 03:25:47.216452 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.215972    2580 scope.go:117] "RemoveContainer" containerID="a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1"
Sep 09 03:25:47.217648 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.217624    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd22869f-85a0-40e6-b9d8-55b21ae1da6b" containerID="e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8" exitCode=0
Sep 09 03:25:47.217784 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.217691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw" event={"ID":"bd22869f-85a0-40e6-b9d8-55b21ae1da6b","Type":"ContainerDied","Data":"e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8"}
Sep 09 03:25:47.217784 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.217716    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"
Sep 09 03:25:47.217784 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.217745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw" event={"ID":"bd22869f-85a0-40e6-b9d8-55b21ae1da6b","Type":"ContainerDied","Data":"9dbabbf7b345ea61016fa97c5839dd96f841f105e342acce593342b04cb1db4a"}
Sep 09 03:25:47.226095 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.226051    2580 scope.go:117] "RemoveContainer" containerID="a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1"
Sep 09 03:25:47.226361 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:25:47.226330    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1\": container with ID starting with a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1 not found: ID does not exist" containerID="a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1"
Sep 09 03:25:47.226476 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.226357    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1"} err="failed to get container status \"a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1\": rpc error: code = NotFound desc = could not find container \"a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1\": container with ID starting with a1d172345524e3a9fa9c0663ed6febdc4331b06dffcb1826c578a42394f08fa1 not found: ID does not exist"
Sep 09 03:25:47.226476 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.226381    2580 scope.go:117] "RemoveContainer" containerID="e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8"
Sep 09 03:25:47.236222 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.236207    2580 scope.go:117] "RemoveContainer" containerID="e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8"
Sep 09 03:25:47.236495 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:25:47.236474    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8\": container with ID starting with e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8 not found: ID does not exist" containerID="e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8"
Sep 09 03:25:47.236541 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.236506    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8"} err="failed to get container status \"e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8\": rpc error: code = NotFound desc = could not find container \"e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8\": container with ID starting with e856d36751b2cc4b471e6f5f5f723aa6b7c57f61bb70f6acf38eea38832993e8 not found: ID does not exist"
Sep 09 03:25:47.266829 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.266803    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"]
Sep 09 03:25:47.277666 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.277641    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70784b-d6db6bd6b-n8mrj"]
Sep 09 03:25:47.296461 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.296432    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"]
Sep 09 03:25:47.312711 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:47.312682    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70784b-7fd96d695f-d4vsw"]
Sep 09 03:25:48.975320 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:48.975285    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd22869f-85a0-40e6-b9d8-55b21ae1da6b" path="/var/lib/kubelet/pods/bd22869f-85a0-40e6-b9d8-55b21ae1da6b/volumes"
Sep 09 03:25:48.975801 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:48.975783    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5173d2d-efe2-42b8-b25c-b2f0b5c46857" path="/var/lib/kubelet/pods/f5173d2d-efe2-42b8-b25c-b2f0b5c46857/volumes"
Sep 09 03:25:50.109796 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:50.109756    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:25:57.220042 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:57.220013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:25:57.220518 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:57.220080    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:25:59.655673 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.655641    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"]
Sep 09 03:25:59.656082 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.655987    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd22869f-85a0-40e6-b9d8-55b21ae1da6b" containerName="payload-processing"
Sep 09 03:25:59.656082 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.655998    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd22869f-85a0-40e6-b9d8-55b21ae1da6b" containerName="payload-processing"
Sep 09 03:25:59.656082 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.656007    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5173d2d-efe2-42b8-b25c-b2f0b5c46857" containerName="payload-pre-processing"
Sep 09 03:25:59.656082 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.656014    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5173d2d-efe2-42b8-b25c-b2f0b5c46857" containerName="payload-pre-processing"
Sep 09 03:25:59.656082 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.656069    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd22869f-85a0-40e6-b9d8-55b21ae1da6b" containerName="payload-processing"
Sep 09 03:25:59.656082 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.656078    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5173d2d-efe2-42b8-b25c-b2f0b5c46857" containerName="payload-pre-processing"
Sep 09 03:25:59.660602 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.660556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:25:59.664547 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.664522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1a85fa-dockercfg-6lpdz\""
Sep 09 03:25:59.664759 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.664742    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1a85fa\""
Sep 09 03:25:59.685060 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.685022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"]
Sep 09 03:25:59.728925 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.728872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571d961e-00a1-4701-80ee-57161ac83c04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m\" (UID: \"571d961e-00a1-4701-80ee-57161ac83c04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:25:59.728925 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.728927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwkkh\" (UniqueName: \"kubernetes.io/projected/571d961e-00a1-4701-80ee-57161ac83c04-kube-api-access-vwkkh\") pod \"payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m\" (UID: \"571d961e-00a1-4701-80ee-57161ac83c04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:25:59.737023 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.736991    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"]
Sep 09 03:25:59.740498 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.740474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:25:59.763830 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.763801    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"]
Sep 09 03:25:59.830564 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.830522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571d961e-00a1-4701-80ee-57161ac83c04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m\" (UID: \"571d961e-00a1-4701-80ee-57161ac83c04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:25:59.830779 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.830569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwkkh\" (UniqueName: \"kubernetes.io/projected/571d961e-00a1-4701-80ee-57161ac83c04-kube-api-access-vwkkh\") pod \"payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m\" (UID: \"571d961e-00a1-4701-80ee-57161ac83c04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:25:59.830779 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.830627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72ffd\" (UniqueName: \"kubernetes.io/projected/561c0b32-a898-4b89-b602-453f493a1812-kube-api-access-72ffd\") pod \"payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw\" (UID: \"561c0b32-a898-4b89-b602-453f493a1812\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:25:59.830779 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.830670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561c0b32-a898-4b89-b602-453f493a1812-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw\" (UID: \"561c0b32-a898-4b89-b602-453f493a1812\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:25:59.831360 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.831341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571d961e-00a1-4701-80ee-57161ac83c04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m\" (UID: \"571d961e-00a1-4701-80ee-57161ac83c04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:25:59.852796 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.852768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwkkh\" (UniqueName: \"kubernetes.io/projected/571d961e-00a1-4701-80ee-57161ac83c04-kube-api-access-vwkkh\") pod \"payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m\" (UID: \"571d961e-00a1-4701-80ee-57161ac83c04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:25:59.932083 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.931977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72ffd\" (UniqueName: \"kubernetes.io/projected/561c0b32-a898-4b89-b602-453f493a1812-kube-api-access-72ffd\") pod \"payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw\" (UID: \"561c0b32-a898-4b89-b602-453f493a1812\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:25:59.932083 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.932028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561c0b32-a898-4b89-b602-453f493a1812-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw\" (UID: \"561c0b32-a898-4b89-b602-453f493a1812\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:25:59.932753 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.932732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561c0b32-a898-4b89-b602-453f493a1812-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw\" (UID: \"561c0b32-a898-4b89-b602-453f493a1812\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:25:59.941130 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.940955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72ffd\" (UniqueName: \"kubernetes.io/projected/561c0b32-a898-4b89-b602-453f493a1812-kube-api-access-72ffd\") pod \"payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw\" (UID: \"561c0b32-a898-4b89-b602-453f493a1812\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:25:59.970706 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:25:59.970671    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:26:00.051082 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:00.051050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:26:00.243450 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:00.243422    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"]
Sep 09 03:26:00.245420 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:26:00.245397    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571d961e_00a1_4701_80ee_57161ac83c04.slice/crio-237c65a27be8b76b50b73e412da640a24785b6a057174161bc5b9ab38500ca7d WatchSource:0}: Error finding container 237c65a27be8b76b50b73e412da640a24785b6a057174161bc5b9ab38500ca7d: Status 404 returned error can't find the container with id 237c65a27be8b76b50b73e412da640a24785b6a057174161bc5b9ab38500ca7d
Sep 09 03:26:00.283364 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:00.283313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m" event={"ID":"571d961e-00a1-4701-80ee-57161ac83c04","Type":"ContainerStarted","Data":"237c65a27be8b76b50b73e412da640a24785b6a057174161bc5b9ab38500ca7d"}
Sep 09 03:26:00.326151 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:00.326125    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"]
Sep 09 03:26:00.328109 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:26:00.328083    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561c0b32_a898_4b89_b602_453f493a1812.slice/crio-8282857dcb85bc83b8535e647c301c636dbdee109c784e0c7f840b1958f532c0 WatchSource:0}: Error finding container 8282857dcb85bc83b8535e647c301c636dbdee109c784e0c7f840b1958f532c0: Status 404 returned error can't find the container with id 8282857dcb85bc83b8535e647c301c636dbdee109c784e0c7f840b1958f532c0
Sep 09 03:26:01.291106 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:01.291054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw" event={"ID":"561c0b32-a898-4b89-b602-453f493a1812","Type":"ContainerStarted","Data":"1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a"}
Sep 09 03:26:01.291569 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:01.291137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw" event={"ID":"561c0b32-a898-4b89-b602-453f493a1812","Type":"ContainerStarted","Data":"8282857dcb85bc83b8535e647c301c636dbdee109c784e0c7f840b1958f532c0"}
Sep 09 03:26:01.291569 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:01.291170    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:26:01.292792 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:01.292768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m" event={"ID":"571d961e-00a1-4701-80ee-57161ac83c04","Type":"ContainerStarted","Data":"405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2"}
Sep 09 03:26:01.292908 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:01.292824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:26:01.309863 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:01.309812    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw" podStartSLOduration=2.30979664 podStartE2EDuration="2.30979664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:01.308446591 +0000 UTC m=+1204.834761357" watchObservedRunningTime="2026-09-09 03:26:01.30979664 +0000 UTC m=+1204.836111378"
Sep 09 03:26:01.327363 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:01.327306    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m" podStartSLOduration=2.327288181 podStartE2EDuration="2.327288181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:01.32585679 +0000 UTC m=+1204.852171553" watchObservedRunningTime="2026-09-09 03:26:01.327288181 +0000 UTC m=+1204.853602919"
Sep 09 03:26:02.738024 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:02.737994    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:26:12.300436 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:12.300405    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:26:12.300934 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:12.300489    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:26:13.863696 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:13.863656    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"]
Sep 09 03:26:13.867531 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:13.867505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:26:13.871139 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:13.871112    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4c8420\""
Sep 09 03:26:13.871139 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:13.871142    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4c8420-dockercfg-52v79\""
Sep 09 03:26:13.888662 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:13.888633    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"]
Sep 09 03:26:13.955974 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:13.955934    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"]
Sep 09 03:26:13.959295 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:13.959274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:26:13.962380 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:13.962338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m657f\" (UniqueName: \"kubernetes.io/projected/50f78d04-56ed-4bc8-8df8-f18a243d4b00-kube-api-access-m657f\") pod \"payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t\" (UID: \"50f78d04-56ed-4bc8-8df8-f18a243d4b00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:26:13.962503 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:13.962449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f78d04-56ed-4bc8-8df8-f18a243d4b00-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t\" (UID: \"50f78d04-56ed-4bc8-8df8-f18a243d4b00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:26:13.999818 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:13.999735    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"]
Sep 09 03:26:14.063973 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.063941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f78d04-56ed-4bc8-8df8-f18a243d4b00-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t\" (UID: \"50f78d04-56ed-4bc8-8df8-f18a243d4b00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:26:14.064185 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.064013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c3af93-5b53-4cb5-a0e8-9ba746113e21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t\" (UID: \"77c3af93-5b53-4cb5-a0e8-9ba746113e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:26:14.064185 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.064095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5zf5\" (UniqueName: \"kubernetes.io/projected/77c3af93-5b53-4cb5-a0e8-9ba746113e21-kube-api-access-n5zf5\") pod \"payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t\" (UID: \"77c3af93-5b53-4cb5-a0e8-9ba746113e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:26:14.064185 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.064138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m657f\" (UniqueName: \"kubernetes.io/projected/50f78d04-56ed-4bc8-8df8-f18a243d4b00-kube-api-access-m657f\") pod \"payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t\" (UID: \"50f78d04-56ed-4bc8-8df8-f18a243d4b00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:26:14.064881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.064848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f78d04-56ed-4bc8-8df8-f18a243d4b00-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t\" (UID: \"50f78d04-56ed-4bc8-8df8-f18a243d4b00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:26:14.081919 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.081886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m657f\" (UniqueName: \"kubernetes.io/projected/50f78d04-56ed-4bc8-8df8-f18a243d4b00-kube-api-access-m657f\") pod \"payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t\" (UID: \"50f78d04-56ed-4bc8-8df8-f18a243d4b00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:26:14.165682 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.165597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c3af93-5b53-4cb5-a0e8-9ba746113e21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t\" (UID: \"77c3af93-5b53-4cb5-a0e8-9ba746113e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:26:14.165682 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.165669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5zf5\" (UniqueName: \"kubernetes.io/projected/77c3af93-5b53-4cb5-a0e8-9ba746113e21-kube-api-access-n5zf5\") pod \"payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t\" (UID: \"77c3af93-5b53-4cb5-a0e8-9ba746113e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:26:14.166201 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.166181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c3af93-5b53-4cb5-a0e8-9ba746113e21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t\" (UID: \"77c3af93-5b53-4cb5-a0e8-9ba746113e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:26:14.175300 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.175268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5zf5\" (UniqueName: \"kubernetes.io/projected/77c3af93-5b53-4cb5-a0e8-9ba746113e21-kube-api-access-n5zf5\") pod \"payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t\" (UID: \"77c3af93-5b53-4cb5-a0e8-9ba746113e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:26:14.179102 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.179085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:26:14.271858 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.271823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:26:14.461151 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.461115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"]
Sep 09 03:26:14.462907 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:26:14.462863    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f78d04_56ed_4bc8_8df8_f18a243d4b00.slice/crio-d8a6d20005c6912a25ea127f9da37ef7262902f4658be8690bfd5a2dfa12ed87 WatchSource:0}: Error finding container d8a6d20005c6912a25ea127f9da37ef7262902f4658be8690bfd5a2dfa12ed87: Status 404 returned error can't find the container with id d8a6d20005c6912a25ea127f9da37ef7262902f4658be8690bfd5a2dfa12ed87
Sep 09 03:26:14.553339 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:14.553310    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"]
Sep 09 03:26:14.555216 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:26:14.555186    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c3af93_5b53_4cb5_a0e8_9ba746113e21.slice/crio-5b952d68014cf8f192f5bcee4ed07b94972b0e87000f6a34f1c4ba620846a135 WatchSource:0}: Error finding container 5b952d68014cf8f192f5bcee4ed07b94972b0e87000f6a34f1c4ba620846a135: Status 404 returned error can't find the container with id 5b952d68014cf8f192f5bcee4ed07b94972b0e87000f6a34f1c4ba620846a135
Sep 09 03:26:15.364437 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:15.364396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t" event={"ID":"77c3af93-5b53-4cb5-a0e8-9ba746113e21","Type":"ContainerStarted","Data":"146cec505056f82fef015b70e5d834a840b8bb7f8a349576b57c8e97b9afe2e4"}
Sep 09 03:26:15.364932 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:15.364494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t" event={"ID":"77c3af93-5b53-4cb5-a0e8-9ba746113e21","Type":"ContainerStarted","Data":"5b952d68014cf8f192f5bcee4ed07b94972b0e87000f6a34f1c4ba620846a135"}
Sep 09 03:26:15.364932 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:15.364527    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:26:15.366080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:15.366054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t" event={"ID":"50f78d04-56ed-4bc8-8df8-f18a243d4b00","Type":"ContainerStarted","Data":"e962d4492cbc54c26d4fde990ae003692f961a71aa8e0098dec22cc575c7b498"}
Sep 09 03:26:15.366216 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:15.366097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t" event={"ID":"50f78d04-56ed-4bc8-8df8-f18a243d4b00","Type":"ContainerStarted","Data":"d8a6d20005c6912a25ea127f9da37ef7262902f4658be8690bfd5a2dfa12ed87"}
Sep 09 03:26:15.366216 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:15.366154    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:26:15.385860 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:15.385794    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t" podStartSLOduration=2.385775082 podStartE2EDuration="2.385775082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:15.38490715 +0000 UTC m=+1218.911221889" watchObservedRunningTime="2026-09-09 03:26:15.385775082 +0000 UTC m=+1218.912089821"
Sep 09 03:26:15.416255 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:15.416212    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t" podStartSLOduration=2.416195094 podStartE2EDuration="2.416195094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:15.415169868 +0000 UTC m=+1218.941484630" watchObservedRunningTime="2026-09-09 03:26:15.416195094 +0000 UTC m=+1218.942509833"
Sep 09 03:26:20.543649 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:20.543603    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:26:26.373495 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:26.373462    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:26:26.373958 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:26.373537    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:26:28.962145 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:28.962100    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"]
Sep 09 03:26:28.965979 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:28.965950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:26:28.972142 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:28.972115    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-cd1da6\""
Sep 09 03:26:28.982501 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:28.982475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"]
Sep 09 03:26:29.104516 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.104454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699\" (UID: \"cc83b8e1-ad9e-4cfd-8c3d-76a048762908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:26:29.104748 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.104724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5hz\" (UniqueName: \"kubernetes.io/projected/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-kube-api-access-8t5hz\") pod \"payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699\" (UID: \"cc83b8e1-ad9e-4cfd-8c3d-76a048762908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:26:29.105997 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.105980    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"]
Sep 09 03:26:29.109535 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.109517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:26:29.112817 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.112797    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-cd1da6-dockercfg-xps4j\""
Sep 09 03:26:29.120744 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.120722    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"]
Sep 09 03:26:29.205350 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.205319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5hz\" (UniqueName: \"kubernetes.io/projected/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-kube-api-access-8t5hz\") pod \"payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699\" (UID: \"cc83b8e1-ad9e-4cfd-8c3d-76a048762908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:26:29.205508 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.205375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699\" (UID: \"cc83b8e1-ad9e-4cfd-8c3d-76a048762908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:26:29.205508 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.205405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxsr\" (UniqueName: \"kubernetes.io/projected/a864a82a-6cbc-453e-9a90-9d7366671883-kube-api-access-6wxsr\") pod \"payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h\" (UID: \"a864a82a-6cbc-453e-9a90-9d7366671883\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:26:29.205508 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.205422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a864a82a-6cbc-453e-9a90-9d7366671883-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h\" (UID: \"a864a82a-6cbc-453e-9a90-9d7366671883\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:26:29.206121 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.206103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699\" (UID: \"cc83b8e1-ad9e-4cfd-8c3d-76a048762908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:26:29.216281 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.216227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5hz\" (UniqueName: \"kubernetes.io/projected/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-kube-api-access-8t5hz\") pod \"payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699\" (UID: \"cc83b8e1-ad9e-4cfd-8c3d-76a048762908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:26:29.276989 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.276948    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:26:29.306445 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.306411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxsr\" (UniqueName: \"kubernetes.io/projected/a864a82a-6cbc-453e-9a90-9d7366671883-kube-api-access-6wxsr\") pod \"payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h\" (UID: \"a864a82a-6cbc-453e-9a90-9d7366671883\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:26:29.306611 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.306455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a864a82a-6cbc-453e-9a90-9d7366671883-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h\" (UID: \"a864a82a-6cbc-453e-9a90-9d7366671883\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:26:29.307390 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.307365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a864a82a-6cbc-453e-9a90-9d7366671883-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h\" (UID: \"a864a82a-6cbc-453e-9a90-9d7366671883\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:26:29.317278 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.317253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxsr\" (UniqueName: \"kubernetes.io/projected/a864a82a-6cbc-453e-9a90-9d7366671883-kube-api-access-6wxsr\") pod \"payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h\" (UID: \"a864a82a-6cbc-453e-9a90-9d7366671883\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:26:29.421066 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.421034    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:26:29.553810 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.553783    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"]
Sep 09 03:26:29.556255 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:26:29.556232    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc83b8e1_ad9e_4cfd_8c3d_76a048762908.slice/crio-a9475cb9554718b009c461c7af51ffeeffb06f1d6073a08bc4c82351df96cb8a WatchSource:0}: Error finding container a9475cb9554718b009c461c7af51ffeeffb06f1d6073a08bc4c82351df96cb8a: Status 404 returned error can't find the container with id a9475cb9554718b009c461c7af51ffeeffb06f1d6073a08bc4c82351df96cb8a
Sep 09 03:26:29.699532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:29.699508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"]
Sep 09 03:26:29.701377 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:26:29.701348    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda864a82a_6cbc_453e_9a90_9d7366671883.slice/crio-1c207ec9c1c7ee11c3877b717923a3a4813425701e1ada3ef1c764fd0dbd6904 WatchSource:0}: Error finding container 1c207ec9c1c7ee11c3877b717923a3a4813425701e1ada3ef1c764fd0dbd6904: Status 404 returned error can't find the container with id 1c207ec9c1c7ee11c3877b717923a3a4813425701e1ada3ef1c764fd0dbd6904
Sep 09 03:26:30.441011 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:30.440971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699" event={"ID":"cc83b8e1-ad9e-4cfd-8c3d-76a048762908","Type":"ContainerStarted","Data":"a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d"}
Sep 09 03:26:30.441477 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:30.441037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699" event={"ID":"cc83b8e1-ad9e-4cfd-8c3d-76a048762908","Type":"ContainerStarted","Data":"a9475cb9554718b009c461c7af51ffeeffb06f1d6073a08bc4c82351df96cb8a"}
Sep 09 03:26:30.441477 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:30.441065    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:26:30.442682 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:30.442654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h" event={"ID":"a864a82a-6cbc-453e-9a90-9d7366671883","Type":"ContainerStarted","Data":"6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d"}
Sep 09 03:26:30.442809 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:30.442704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h" event={"ID":"a864a82a-6cbc-453e-9a90-9d7366671883","Type":"ContainerStarted","Data":"1c207ec9c1c7ee11c3877b717923a3a4813425701e1ada3ef1c764fd0dbd6904"}
Sep 09 03:26:30.442809 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:30.442727    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:26:30.463346 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:30.463303    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699" podStartSLOduration=2.463286856 podStartE2EDuration="2.463286856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:30.461339227 +0000 UTC m=+1233.987653966" watchObservedRunningTime="2026-09-09 03:26:30.463286856 +0000 UTC m=+1233.989601630"
Sep 09 03:26:30.482979 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:30.482927    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h" podStartSLOduration=1.482908925 podStartE2EDuration="1.482908925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:30.481371515 +0000 UTC m=+1234.007686256" watchObservedRunningTime="2026-09-09 03:26:30.482908925 +0000 UTC m=+1234.009223647"
Sep 09 03:26:33.019240 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:33.019203    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:26:41.449762 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:41.449726    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:26:41.450175 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:41.449893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:26:56.289313 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.289276    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"]
Sep 09 03:26:56.292784 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.292767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:26:56.296694 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.296669    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a3a90070-dockercfg-pxxzv\""
Sep 09 03:26:56.296807 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.296708    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a3a90070\""
Sep 09 03:26:56.316964 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.316934    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"]
Sep 09 03:26:56.358641 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.358612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs\" (UID: \"20a69bd9-dce1-47eb-9348-a5f10cecb1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:26:56.358780 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.358673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqblm\" (UniqueName: \"kubernetes.io/projected/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-kube-api-access-hqblm\") pod \"payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs\" (UID: \"20a69bd9-dce1-47eb-9348-a5f10cecb1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:26:56.394531 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.394494    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"]
Sep 09 03:26:56.398132 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.398117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:26:56.414106 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.414081    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"]
Sep 09 03:26:56.459315 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.459281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs\" (UID: \"20a69bd9-dce1-47eb-9348-a5f10cecb1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:26:56.459464 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.459328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-plugins-config-volume\") pod \"payload-processing-e2e-mt-a3a90070-58745888f7-22wmd\" (UID: \"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d\") " pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:26:56.459464 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.459358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqblm\" (UniqueName: \"kubernetes.io/projected/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-kube-api-access-hqblm\") pod \"payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs\" (UID: \"20a69bd9-dce1-47eb-9348-a5f10cecb1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:26:56.459464 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.459401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6mp\" (UniqueName: \"kubernetes.io/projected/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-kube-api-access-vc6mp\") pod \"payload-processing-e2e-mt-a3a90070-58745888f7-22wmd\" (UID: \"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d\") " pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:26:56.460010 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.459990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs\" (UID: \"20a69bd9-dce1-47eb-9348-a5f10cecb1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:26:56.476678 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.476653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqblm\" (UniqueName: \"kubernetes.io/projected/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-kube-api-access-hqblm\") pod \"payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs\" (UID: \"20a69bd9-dce1-47eb-9348-a5f10cecb1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:26:56.560445 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.560376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-plugins-config-volume\") pod \"payload-processing-e2e-mt-a3a90070-58745888f7-22wmd\" (UID: \"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d\") " pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:26:56.560445 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.560432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6mp\" (UniqueName: \"kubernetes.io/projected/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-kube-api-access-vc6mp\") pod \"payload-processing-e2e-mt-a3a90070-58745888f7-22wmd\" (UID: \"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d\") " pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:26:56.561082 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.561063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-plugins-config-volume\") pod \"payload-processing-e2e-mt-a3a90070-58745888f7-22wmd\" (UID: \"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d\") " pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:26:56.569953 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.569931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6mp\" (UniqueName: \"kubernetes.io/projected/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-kube-api-access-vc6mp\") pod \"payload-processing-e2e-mt-a3a90070-58745888f7-22wmd\" (UID: \"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d\") " pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:26:56.603030 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.603011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:26:56.708529 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.708499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:26:56.903286 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.903254    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"]
Sep 09 03:26:56.909802 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:26:56.909774    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a69bd9_dce1_47eb_9348_a5f10cecb1f2.slice/crio-960c2547276141b3e3c1cb4c7195f35823ac561ee7d8b65e842c3639d98cfa1a WatchSource:0}: Error finding container 960c2547276141b3e3c1cb4c7195f35823ac561ee7d8b65e842c3639d98cfa1a: Status 404 returned error can't find the container with id 960c2547276141b3e3c1cb4c7195f35823ac561ee7d8b65e842c3639d98cfa1a
Sep 09 03:26:56.994827 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:56.994732    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"]
Sep 09 03:26:56.999531 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:26:56.999491    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07439ca_185f_4cbc_9fe3_5a3efe0e5a2d.slice/crio-7e818259cadfcf4e43b71ad981623427f4e178f9e263cf86b85c42c1dffb9efb WatchSource:0}: Error finding container 7e818259cadfcf4e43b71ad981623427f4e178f9e263cf86b85c42c1dffb9efb: Status 404 returned error can't find the container with id 7e818259cadfcf4e43b71ad981623427f4e178f9e263cf86b85c42c1dffb9efb
Sep 09 03:26:57.586599 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:57.586539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd" event={"ID":"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d","Type":"ContainerStarted","Data":"2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b"}
Sep 09 03:26:57.587077 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:57.586670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd" event={"ID":"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d","Type":"ContainerStarted","Data":"7e818259cadfcf4e43b71ad981623427f4e178f9e263cf86b85c42c1dffb9efb"}
Sep 09 03:26:57.587077 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:57.586705    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:26:57.588332 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:57.588296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs" event={"ID":"20a69bd9-dce1-47eb-9348-a5f10cecb1f2","Type":"ContainerStarted","Data":"0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179"}
Sep 09 03:26:57.588467 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:57.588346    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs" event={"ID":"20a69bd9-dce1-47eb-9348-a5f10cecb1f2","Type":"ContainerStarted","Data":"960c2547276141b3e3c1cb4c7195f35823ac561ee7d8b65e842c3639d98cfa1a"}
Sep 09 03:26:57.588467 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:57.588388    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:26:57.611193 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:57.611137    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd" podStartSLOduration=1.6111195710000001 podStartE2EDuration="1.611119571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:57.609751615 +0000 UTC m=+1261.136066355" watchObservedRunningTime="2026-09-09 03:26:57.611119571 +0000 UTC m=+1261.137434311"
Sep 09 03:26:57.630772 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:26:57.630712    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs" podStartSLOduration=1.6306929000000001 podStartE2EDuration="1.6306929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:57.630400458 +0000 UTC m=+1261.156715197" watchObservedRunningTime="2026-09-09 03:26:57.6306929 +0000 UTC m=+1261.157007640"
Sep 09 03:27:02.724153 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:27:02.724112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:27:08.596361 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:27:08.596331    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:27:08.596744 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:27:08.596517    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:27:19.570452 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:27:19.570414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:27:41.705247 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:27:41.705212    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:28:01.722204 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:01.722166    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:28:20.724144 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:20.724049    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:28:39.410825 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:39.410783    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:28:40.223150 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.223120    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"]
Sep 09 03:28:40.223360 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.223334    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd" podUID="f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d" containerName="payload-processing" containerID="cri-o://2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b" gracePeriod=30
Sep 09 03:28:40.232052 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.232024    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"]
Sep 09 03:28:40.232250 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.232227    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs" podUID="20a69bd9-dce1-47eb-9348-a5f10cecb1f2" containerName="payload-pre-processing" containerID="cri-o://0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179" gracePeriod=30
Sep 09 03:28:40.499719 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.499694    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:28:40.529675 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.529651    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:28:40.620168 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.620138    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqblm\" (UniqueName: \"kubernetes.io/projected/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-kube-api-access-hqblm\") pod \"20a69bd9-dce1-47eb-9348-a5f10cecb1f2\" (UID: \"20a69bd9-dce1-47eb-9348-a5f10cecb1f2\") "
Sep 09 03:28:40.620368 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.620207    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc6mp\" (UniqueName: \"kubernetes.io/projected/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-kube-api-access-vc6mp\") pod \"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d\" (UID: \"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d\") "
Sep 09 03:28:40.620368 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.620290    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-plugins-config-volume\") pod \"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d\" (UID: \"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d\") "
Sep 09 03:28:40.620368 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.620329    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-plugins-config-volume\") pod \"20a69bd9-dce1-47eb-9348-a5f10cecb1f2\" (UID: \"20a69bd9-dce1-47eb-9348-a5f10cecb1f2\") "
Sep 09 03:28:40.620705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.620672    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d" (UID: "f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:28:40.620828 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.620769    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20a69bd9-dce1-47eb-9348-a5f10cecb1f2" (UID: "20a69bd9-dce1-47eb-9348-a5f10cecb1f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:28:40.622494 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.622471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-kube-api-access-vc6mp" (OuterVolumeSpecName: "kube-api-access-vc6mp") pod "f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d" (UID: "f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d"). InnerVolumeSpecName "kube-api-access-vc6mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:28:40.622627 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.622516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-kube-api-access-hqblm" (OuterVolumeSpecName: "kube-api-access-hqblm") pod "20a69bd9-dce1-47eb-9348-a5f10cecb1f2" (UID: "20a69bd9-dce1-47eb-9348-a5f10cecb1f2"). InnerVolumeSpecName "kube-api-access-hqblm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:28:40.721600 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.721557    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:28:40.721753 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.721614    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:28:40.721753 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.721630    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqblm\" (UniqueName: \"kubernetes.io/projected/20a69bd9-dce1-47eb-9348-a5f10cecb1f2-kube-api-access-hqblm\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:28:40.721753 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:40.721643    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc6mp\" (UniqueName: \"kubernetes.io/projected/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d-kube-api-access-vc6mp\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:28:41.113967 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.113934    2580 generic.go:356] "Generic (PLEG): container finished" podID="f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d" containerID="2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b" exitCode=0
Sep 09 03:28:41.114178 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.113992    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"
Sep 09 03:28:41.114178 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.114025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd" event={"ID":"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d","Type":"ContainerDied","Data":"2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b"}
Sep 09 03:28:41.114178 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.114077    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd" event={"ID":"f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d","Type":"ContainerDied","Data":"7e818259cadfcf4e43b71ad981623427f4e178f9e263cf86b85c42c1dffb9efb"}
Sep 09 03:28:41.114178 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.114104    2580 scope.go:117] "RemoveContainer" containerID="2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b"
Sep 09 03:28:41.115488 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.115465    2580 generic.go:356] "Generic (PLEG): container finished" podID="20a69bd9-dce1-47eb-9348-a5f10cecb1f2" containerID="0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179" exitCode=0
Sep 09 03:28:41.115623 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.115516    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"
Sep 09 03:28:41.115623 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.115565    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs" event={"ID":"20a69bd9-dce1-47eb-9348-a5f10cecb1f2","Type":"ContainerDied","Data":"0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179"}
Sep 09 03:28:41.115738 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.115624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs" event={"ID":"20a69bd9-dce1-47eb-9348-a5f10cecb1f2","Type":"ContainerDied","Data":"960c2547276141b3e3c1cb4c7195f35823ac561ee7d8b65e842c3639d98cfa1a"}
Sep 09 03:28:41.123415 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.123399    2580 scope.go:117] "RemoveContainer" containerID="2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b"
Sep 09 03:28:41.123755 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:28:41.123735    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b\": container with ID starting with 2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b not found: ID does not exist" containerID="2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b"
Sep 09 03:28:41.123819 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.123763    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b"} err="failed to get container status \"2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b\": rpc error: code = NotFound desc = could not find container \"2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b\": container with ID starting with 2a0d716c60a58c595f74311b57a7a1fa413b696c513ea87748f1be9c512f142b not found: ID does not exist"
Sep 09 03:28:41.123819 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.123781    2580 scope.go:117] "RemoveContainer" containerID="0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179"
Sep 09 03:28:41.132421 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.132402    2580 scope.go:117] "RemoveContainer" containerID="0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179"
Sep 09 03:28:41.132801 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:28:41.132779    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179\": container with ID starting with 0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179 not found: ID does not exist" containerID="0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179"
Sep 09 03:28:41.132947 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.132808    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179"} err="failed to get container status \"0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179\": rpc error: code = NotFound desc = could not find container \"0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179\": container with ID starting with 0c2d70ae6892d7cdbf35cf46e09cc8f55f6d4a9338c2c11e4d05a4581fe07179 not found: ID does not exist"
Sep 09 03:28:41.133426 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.133407    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"]
Sep 09 03:28:41.137522 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.137502    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3a90070-8595cc7b74-mn2zs"]
Sep 09 03:28:41.151175 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.151147    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"]
Sep 09 03:28:41.156207 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:41.156181    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3a90070-58745888f7-22wmd"]
Sep 09 03:28:42.971624 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:42.971572    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a69bd9-dce1-47eb-9348-a5f10cecb1f2" path="/var/lib/kubelet/pods/20a69bd9-dce1-47eb-9348-a5f10cecb1f2/volumes"
Sep 09 03:28:42.972022 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:42.972003    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d" path="/var/lib/kubelet/pods/f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d/volumes"
Sep 09 03:28:57.057236 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:57.057197    2580 scope.go:117] "RemoveContainer" containerID="266ff27ebb2e373be55f4c56061c99edac1fd9975f48165ec2635ec4f02d9d6b"
Sep 09 03:28:57.925443 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:28:57.925408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:29:14.612896 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:29:14.612858    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:29:31.744780 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:29:31.744744    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:29:48.417264 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:29:48.417229    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:30:00.149777 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.149741    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm"]
Sep 09 03:30:00.150240 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.150151    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d" containerName="payload-processing"
Sep 09 03:30:00.150240 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.150163    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d" containerName="payload-processing"
Sep 09 03:30:00.150240 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.150174    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a69bd9-dce1-47eb-9348-a5f10cecb1f2" containerName="payload-pre-processing"
Sep 09 03:30:00.150240 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.150179    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a69bd9-dce1-47eb-9348-a5f10cecb1f2" containerName="payload-pre-processing"
Sep 09 03:30:00.150240 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.150238    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f07439ca-185f-4cbc-9fe3-5a3efe0e5a2d" containerName="payload-processing"
Sep 09 03:30:00.150403 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.150247    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a69bd9-dce1-47eb-9348-a5f10cecb1f2" containerName="payload-pre-processing"
Sep 09 03:30:00.153402 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.153375    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm"
Sep 09 03:30:00.155027 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.155006    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815410-qdrr5"]
Sep 09 03:30:00.156323 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.156297    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:30:00.158702 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.158685    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc"]
Sep 09 03:30:00.158883 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.158853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815410-qdrr5"
Sep 09 03:30:00.162178 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.162156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc"
Sep 09 03:30:00.164545 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.164530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm"
Sep 09 03:30:00.172731 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.172706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815410-qdrr5"
Sep 09 03:30:00.174368 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.174340    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm"]
Sep 09 03:30:00.179635 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.179608    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815410-qdrr5"]
Sep 09 03:30:00.182755 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.182730    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc"]
Sep 09 03:30:00.210890 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.210849    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc"
Sep 09 03:30:00.455287 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.455264    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm"]
Sep 09 03:30:00.457554 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:30:00.457530    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b33b87_e546_4134_a382_09c78364f772.slice/crio-902e47144c36530ed09f356a4344ab521f9b813bfb29eb16418ea624d6b6a4fd WatchSource:0}: Error finding container 902e47144c36530ed09f356a4344ab521f9b813bfb29eb16418ea624d6b6a4fd: Status 404 returned error can't find the container with id 902e47144c36530ed09f356a4344ab521f9b813bfb29eb16418ea624d6b6a4fd
Sep 09 03:30:00.463378 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.463336    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815410-qdrr5"]
Sep 09 03:30:00.465531 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:30:00.465501    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2502ad_9a26_436f_8b5b_41dc0fd3a35d.slice/crio-3882f6d62b51cc1c0c7a2ae2c9f3ad48e4e15595086160cbe32822f9148036c1 WatchSource:0}: Error finding container 3882f6d62b51cc1c0c7a2ae2c9f3ad48e4e15595086160cbe32822f9148036c1: Status 404 returned error can't find the container with id 3882f6d62b51cc1c0c7a2ae2c9f3ad48e4e15595086160cbe32822f9148036c1
Sep 09 03:30:00.503869 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.503840    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc"]
Sep 09 03:30:00.506422 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:30:00.506390    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e313e8_0f09_4809_b310_dcfde8b33915.slice/crio-758f5e18d4b395272d382530d0b11a23a3cd1dfdd934c4ae5980f21e7af59c2e WatchSource:0}: Error finding container 758f5e18d4b395272d382530d0b11a23a3cd1dfdd934c4ae5980f21e7af59c2e: Status 404 returned error can't find the container with id 758f5e18d4b395272d382530d0b11a23a3cd1dfdd934c4ae5980f21e7af59c2e
Sep 09 03:30:00.517813 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.517783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc" event={"ID":"83e313e8-0f09-4809-b310-dcfde8b33915","Type":"ContainerStarted","Data":"758f5e18d4b395272d382530d0b11a23a3cd1dfdd934c4ae5980f21e7af59c2e"}
Sep 09 03:30:00.519026 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.519002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm" event={"ID":"a2b33b87-e546-4134-a382-09c78364f772","Type":"ContainerStarted","Data":"902e47144c36530ed09f356a4344ab521f9b813bfb29eb16418ea624d6b6a4fd"}
Sep 09 03:30:00.520131 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:00.520103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815410-qdrr5" event={"ID":"0c2502ad-9a26-436f-8b5b-41dc0fd3a35d","Type":"ContainerStarted","Data":"3882f6d62b51cc1c0c7a2ae2c9f3ad48e4e15595086160cbe32822f9148036c1"}
Sep 09 03:30:01.527417 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:01.527380    2580 generic.go:356] "Generic (PLEG): container finished" podID="83e313e8-0f09-4809-b310-dcfde8b33915" containerID="3628f627109d638488d5903151efc964b97432017e99beaeda20b0f3a76321ad" exitCode=0
Sep 09 03:30:01.527905 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:01.527465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc" event={"ID":"83e313e8-0f09-4809-b310-dcfde8b33915","Type":"ContainerDied","Data":"3628f627109d638488d5903151efc964b97432017e99beaeda20b0f3a76321ad"}
Sep 09 03:30:01.529210 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:01.529180    2580 generic.go:356] "Generic (PLEG): container finished" podID="a2b33b87-e546-4134-a382-09c78364f772" containerID="216036402a4eab381c8d649ccefdca14ac9a1644413a2266271e696edfd4a803" exitCode=0
Sep 09 03:30:01.529334 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:01.529266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm" event={"ID":"a2b33b87-e546-4134-a382-09c78364f772","Type":"ContainerDied","Data":"216036402a4eab381c8d649ccefdca14ac9a1644413a2266271e696edfd4a803"}
Sep 09 03:30:01.530967 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:01.530948    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c2502ad-9a26-436f-8b5b-41dc0fd3a35d" containerID="0be7c06e4187fea1933984a29bd951214f12bce992d109ad4b3b114937df0a63" exitCode=0
Sep 09 03:30:01.531056 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:01.531013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815410-qdrr5" event={"ID":"0c2502ad-9a26-436f-8b5b-41dc0fd3a35d","Type":"ContainerDied","Data":"0be7c06e4187fea1933984a29bd951214f12bce992d109ad4b3b114937df0a63"}
Sep 09 03:30:02.670337 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:02.670297    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815410-qdrr5"
Sep 09 03:30:02.700710 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:02.700687    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc"
Sep 09 03:30:02.715797 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:02.715768    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm"
Sep 09 03:30:03.541946 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:03.541906    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc"
Sep 09 03:30:03.542140 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:03.541912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc" event={"ID":"83e313e8-0f09-4809-b310-dcfde8b33915","Type":"ContainerDied","Data":"758f5e18d4b395272d382530d0b11a23a3cd1dfdd934c4ae5980f21e7af59c2e"}
Sep 09 03:30:03.542140 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:03.542062    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758f5e18d4b395272d382530d0b11a23a3cd1dfdd934c4ae5980f21e7af59c2e"
Sep 09 03:30:03.543377 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:03.543353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm" event={"ID":"a2b33b87-e546-4134-a382-09c78364f772","Type":"ContainerDied","Data":"902e47144c36530ed09f356a4344ab521f9b813bfb29eb16418ea624d6b6a4fd"}
Sep 09 03:30:03.543517 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:03.543396    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="902e47144c36530ed09f356a4344ab521f9b813bfb29eb16418ea624d6b6a4fd"
Sep 09 03:30:03.543517 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:03.543362    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm"
Sep 09 03:30:03.544842 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:03.544824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815410-qdrr5" event={"ID":"0c2502ad-9a26-436f-8b5b-41dc0fd3a35d","Type":"ContainerDied","Data":"3882f6d62b51cc1c0c7a2ae2c9f3ad48e4e15595086160cbe32822f9148036c1"}
Sep 09 03:30:03.544926 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:03.544856    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3882f6d62b51cc1c0c7a2ae2c9f3ad48e4e15595086160cbe32822f9148036c1"
Sep 09 03:30:03.544926 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:03.544829    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815410-qdrr5"
Sep 09 03:30:03.700558 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:03.700523    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:30:19.012810 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:19.012774    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:30:32.000250 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.000211    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"]
Sep 09 03:30:32.000646 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.000625    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b33b87-e546-4134-a382-09c78364f772" containerName="cleanup"
Sep 09 03:30:32.000646 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.000637    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b33b87-e546-4134-a382-09c78364f772" containerName="cleanup"
Sep 09 03:30:32.000728 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.000652    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e313e8-0f09-4809-b310-dcfde8b33915" containerName="cleanup"
Sep 09 03:30:32.000728 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.000657    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e313e8-0f09-4809-b310-dcfde8b33915" containerName="cleanup"
Sep 09 03:30:32.000728 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.000664    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2502ad-9a26-436f-8b5b-41dc0fd3a35d" containerName="cleanup"
Sep 09 03:30:32.000728 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.000670    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2502ad-9a26-436f-8b5b-41dc0fd3a35d" containerName="cleanup"
Sep 09 03:30:32.000852 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.000731    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e313e8-0f09-4809-b310-dcfde8b33915" containerName="cleanup"
Sep 09 03:30:32.000852 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.000739    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2502ad-9a26-436f-8b5b-41dc0fd3a35d" containerName="cleanup"
Sep 09 03:30:32.000852 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.000747    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b33b87-e546-4134-a382-09c78364f772" containerName="cleanup"
Sep 09 03:30:32.003838 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.003817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:30:32.006642 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.006622    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ec5029\""
Sep 09 03:30:32.006743 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.006654    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ec5029-dockercfg-wpq2t\""
Sep 09 03:30:32.014759 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.014738    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"]
Sep 09 03:30:32.020752 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.020730    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"]
Sep 09 03:30:32.026299 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.026278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:30:32.040353 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.040335    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"]
Sep 09 03:30:32.160336 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.160298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttf7\" (UniqueName: \"kubernetes.io/projected/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-kube-api-access-tttf7\") pod \"payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf\" (UID: \"c9dd2645-6ca1-46d3-9e2a-a618bad8447b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:30:32.160336 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.160338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvqcn\" (UniqueName: \"kubernetes.io/projected/b5eeb8b5-2f76-4e63-b503-14868ba34a26-kube-api-access-dvqcn\") pod \"payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w\" (UID: \"b5eeb8b5-2f76-4e63-b503-14868ba34a26\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:30:32.160648 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.160363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5eeb8b5-2f76-4e63-b503-14868ba34a26-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w\" (UID: \"b5eeb8b5-2f76-4e63-b503-14868ba34a26\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:30:32.160648 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.160489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf\" (UID: \"c9dd2645-6ca1-46d3-9e2a-a618bad8447b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:30:32.261489 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.261389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5eeb8b5-2f76-4e63-b503-14868ba34a26-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w\" (UID: \"b5eeb8b5-2f76-4e63-b503-14868ba34a26\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:30:32.261489 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.261472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf\" (UID: \"c9dd2645-6ca1-46d3-9e2a-a618bad8447b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:30:32.261732 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.261570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttf7\" (UniqueName: \"kubernetes.io/projected/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-kube-api-access-tttf7\") pod \"payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf\" (UID: \"c9dd2645-6ca1-46d3-9e2a-a618bad8447b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:30:32.261732 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.261628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvqcn\" (UniqueName: \"kubernetes.io/projected/b5eeb8b5-2f76-4e63-b503-14868ba34a26-kube-api-access-dvqcn\") pod \"payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w\" (UID: \"b5eeb8b5-2f76-4e63-b503-14868ba34a26\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:30:32.262238 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.262218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5eeb8b5-2f76-4e63-b503-14868ba34a26-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w\" (UID: \"b5eeb8b5-2f76-4e63-b503-14868ba34a26\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:30:32.262302 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.262255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf\" (UID: \"c9dd2645-6ca1-46d3-9e2a-a618bad8447b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:30:32.270341 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.270313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvqcn\" (UniqueName: \"kubernetes.io/projected/b5eeb8b5-2f76-4e63-b503-14868ba34a26-kube-api-access-dvqcn\") pod \"payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w\" (UID: \"b5eeb8b5-2f76-4e63-b503-14868ba34a26\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:30:32.270462 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.270411    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttf7\" (UniqueName: \"kubernetes.io/projected/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-kube-api-access-tttf7\") pod \"payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf\" (UID: \"c9dd2645-6ca1-46d3-9e2a-a618bad8447b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:30:32.318954 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.318924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:30:32.335851 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.335822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:30:32.597916 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.597865    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"]
Sep 09 03:30:32.600285 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:30:32.600257    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9dd2645_6ca1_46d3_9e2a_a618bad8447b.slice/crio-7c0780c9b64f610831d6ec2842971b0b487643e30a7e0875281dbf5acd78f52c WatchSource:0}: Error finding container 7c0780c9b64f610831d6ec2842971b0b487643e30a7e0875281dbf5acd78f52c: Status 404 returned error can't find the container with id 7c0780c9b64f610831d6ec2842971b0b487643e30a7e0875281dbf5acd78f52c
Sep 09 03:30:32.623721 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.623693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"]
Sep 09 03:30:32.636768 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:30:32.636741    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5eeb8b5_2f76_4e63_b503_14868ba34a26.slice/crio-ee117f7cc4917ea2bc6ce08af959f95b6bf0a89439875c75d82518e3c056e665 WatchSource:0}: Error finding container ee117f7cc4917ea2bc6ce08af959f95b6bf0a89439875c75d82518e3c056e665: Status 404 returned error can't find the container with id ee117f7cc4917ea2bc6ce08af959f95b6bf0a89439875c75d82518e3c056e665
Sep 09 03:30:32.688656 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.688626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf" event={"ID":"c9dd2645-6ca1-46d3-9e2a-a618bad8447b","Type":"ContainerStarted","Data":"cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd"}
Sep 09 03:30:32.688797 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.688688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf" event={"ID":"c9dd2645-6ca1-46d3-9e2a-a618bad8447b","Type":"ContainerStarted","Data":"7c0780c9b64f610831d6ec2842971b0b487643e30a7e0875281dbf5acd78f52c"}
Sep 09 03:30:32.688797 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.688717    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:30:32.690380 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.690349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w" event={"ID":"b5eeb8b5-2f76-4e63-b503-14868ba34a26","Type":"ContainerStarted","Data":"ee117f7cc4917ea2bc6ce08af959f95b6bf0a89439875c75d82518e3c056e665"}
Sep 09 03:30:32.707339 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.707279    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf" podStartSLOduration=1.707261438 podStartE2EDuration="1.707261438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:32.706371488 +0000 UTC m=+1476.232686226" watchObservedRunningTime="2026-09-09 03:30:32.707261438 +0000 UTC m=+1476.233576177"
Sep 09 03:30:32.948380 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:32.948343    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:30:33.697299 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:33.697261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w" event={"ID":"b5eeb8b5-2f76-4e63-b503-14868ba34a26","Type":"ContainerStarted","Data":"c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d"}
Sep 09 03:30:33.697735 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:33.697457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:30:33.722710 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:33.717132    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w" podStartSLOduration=2.717109903 podStartE2EDuration="2.717109903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:33.714344965 +0000 UTC m=+1477.240659705" watchObservedRunningTime="2026-09-09 03:30:33.717109903 +0000 UTC m=+1477.243424646"
Sep 09 03:30:43.700119 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:43.700086    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:30:44.704760 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:44.704732    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:30:48.714746 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:30:48.714715    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:31:01.266963 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.266925    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"]
Sep 09 03:31:01.270443 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.270422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:31:01.273372 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.273350    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-811c96-dockercfg-zbsc6\""
Sep 09 03:31:01.273699 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.273683    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-811c96\""
Sep 09 03:31:01.284136 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.284113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"]
Sep 09 03:31:01.345667 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.345626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfdsw\" (UniqueName: \"kubernetes.io/projected/d1c7d912-c0f5-40a7-8e89-f4953e060871-kube-api-access-tfdsw\") pod \"payload-processing-e2e-api-b-811c96-6cd859645-5hvxf\" (UID: \"d1c7d912-c0f5-40a7-8e89-f4953e060871\") " pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:31:01.345891 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.345781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c7d912-c0f5-40a7-8e89-f4953e060871-plugins-config-volume\") pod \"payload-processing-e2e-api-b-811c96-6cd859645-5hvxf\" (UID: \"d1c7d912-c0f5-40a7-8e89-f4953e060871\") " pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:31:01.446649 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.446615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfdsw\" (UniqueName: \"kubernetes.io/projected/d1c7d912-c0f5-40a7-8e89-f4953e060871-kube-api-access-tfdsw\") pod \"payload-processing-e2e-api-b-811c96-6cd859645-5hvxf\" (UID: \"d1c7d912-c0f5-40a7-8e89-f4953e060871\") " pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:31:01.446842 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.446677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c7d912-c0f5-40a7-8e89-f4953e060871-plugins-config-volume\") pod \"payload-processing-e2e-api-b-811c96-6cd859645-5hvxf\" (UID: \"d1c7d912-c0f5-40a7-8e89-f4953e060871\") " pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:31:01.447329 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.447310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c7d912-c0f5-40a7-8e89-f4953e060871-plugins-config-volume\") pod \"payload-processing-e2e-api-b-811c96-6cd859645-5hvxf\" (UID: \"d1c7d912-c0f5-40a7-8e89-f4953e060871\") " pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:31:01.455135 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.455108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfdsw\" (UniqueName: \"kubernetes.io/projected/d1c7d912-c0f5-40a7-8e89-f4953e060871-kube-api-access-tfdsw\") pod \"payload-processing-e2e-api-b-811c96-6cd859645-5hvxf\" (UID: \"d1c7d912-c0f5-40a7-8e89-f4953e060871\") " pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:31:01.581213 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.581128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:31:01.855638 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:01.855605    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"]
Sep 09 03:31:01.858725 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:31:01.858702    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c7d912_c0f5_40a7_8e89_f4953e060871.slice/crio-c948c22cb0f5bb426edd87995451dd12cbc181b18a7a954f26b8a3fd547914de WatchSource:0}: Error finding container c948c22cb0f5bb426edd87995451dd12cbc181b18a7a954f26b8a3fd547914de: Status 404 returned error can't find the container with id c948c22cb0f5bb426edd87995451dd12cbc181b18a7a954f26b8a3fd547914de
Sep 09 03:31:02.849950 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:02.849915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf" event={"ID":"d1c7d912-c0f5-40a7-8e89-f4953e060871","Type":"ContainerStarted","Data":"b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854"}
Sep 09 03:31:02.850401 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:02.849965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf" event={"ID":"d1c7d912-c0f5-40a7-8e89-f4953e060871","Type":"ContainerStarted","Data":"c948c22cb0f5bb426edd87995451dd12cbc181b18a7a954f26b8a3fd547914de"}
Sep 09 03:31:02.850401 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:02.850030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:31:02.868863 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:02.868813    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf" podStartSLOduration=1.8687950500000001 podStartE2EDuration="1.86879505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:31:02.866859803 +0000 UTC m=+1506.393174543" watchObservedRunningTime="2026-09-09 03:31:02.86879505 +0000 UTC m=+1506.395109789"
Sep 09 03:31:05.206086 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:05.206049    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:31:13.857609 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:13.857564    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:31:27.713011 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:27.712975    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:31:39.909736 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:39.909704    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:31:42.973480 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:42.973451    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"]
Sep 09 03:31:42.973995 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:42.973693    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w" podUID="b5eeb8b5-2f76-4e63-b503-14868ba34a26" containerName="payload-processing" containerID="cri-o://c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d" gracePeriod=30
Sep 09 03:31:42.976980 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:42.976953    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"]
Sep 09 03:31:42.977879 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:42.977851    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf" podUID="c9dd2645-6ca1-46d3-9e2a-a618bad8447b" containerName="payload-pre-processing" containerID="cri-o://cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd" gracePeriod=30
Sep 09 03:31:43.264463 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.264444    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:31:43.267548 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.267529    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:31:43.348870 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.348839    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tttf7\" (UniqueName: \"kubernetes.io/projected/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-kube-api-access-tttf7\") pod \"c9dd2645-6ca1-46d3-9e2a-a618bad8447b\" (UID: \"c9dd2645-6ca1-46d3-9e2a-a618bad8447b\") "
Sep 09 03:31:43.349043 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.348941    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-plugins-config-volume\") pod \"c9dd2645-6ca1-46d3-9e2a-a618bad8447b\" (UID: \"c9dd2645-6ca1-46d3-9e2a-a618bad8447b\") "
Sep 09 03:31:43.349043 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.348999    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5eeb8b5-2f76-4e63-b503-14868ba34a26-plugins-config-volume\") pod \"b5eeb8b5-2f76-4e63-b503-14868ba34a26\" (UID: \"b5eeb8b5-2f76-4e63-b503-14868ba34a26\") "
Sep 09 03:31:43.349115 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.349048    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvqcn\" (UniqueName: \"kubernetes.io/projected/b5eeb8b5-2f76-4e63-b503-14868ba34a26-kube-api-access-dvqcn\") pod \"b5eeb8b5-2f76-4e63-b503-14868ba34a26\" (UID: \"b5eeb8b5-2f76-4e63-b503-14868ba34a26\") "
Sep 09 03:31:43.349306 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.349275    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9dd2645-6ca1-46d3-9e2a-a618bad8447b" (UID: "c9dd2645-6ca1-46d3-9e2a-a618bad8447b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:31:43.349443 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.349375    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:31:43.349443 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.349402    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5eeb8b5-2f76-4e63-b503-14868ba34a26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5eeb8b5-2f76-4e63-b503-14868ba34a26" (UID: "b5eeb8b5-2f76-4e63-b503-14868ba34a26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:31:43.351129 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.351108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-kube-api-access-tttf7" (OuterVolumeSpecName: "kube-api-access-tttf7") pod "c9dd2645-6ca1-46d3-9e2a-a618bad8447b" (UID: "c9dd2645-6ca1-46d3-9e2a-a618bad8447b"). InnerVolumeSpecName "kube-api-access-tttf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:31:43.351215 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.351196    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5eeb8b5-2f76-4e63-b503-14868ba34a26-kube-api-access-dvqcn" (OuterVolumeSpecName: "kube-api-access-dvqcn") pod "b5eeb8b5-2f76-4e63-b503-14868ba34a26" (UID: "b5eeb8b5-2f76-4e63-b503-14868ba34a26"). InnerVolumeSpecName "kube-api-access-dvqcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:31:43.450662 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.450626    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tttf7\" (UniqueName: \"kubernetes.io/projected/c9dd2645-6ca1-46d3-9e2a-a618bad8447b-kube-api-access-tttf7\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:31:43.450662 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.450653    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5eeb8b5-2f76-4e63-b503-14868ba34a26-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:31:43.450662 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:43.450662    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvqcn\" (UniqueName: \"kubernetes.io/projected/b5eeb8b5-2f76-4e63-b503-14868ba34a26-kube-api-access-dvqcn\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:31:44.073448 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.073409    2580 generic.go:356] "Generic (PLEG): container finished" podID="c9dd2645-6ca1-46d3-9e2a-a618bad8447b" containerID="cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd" exitCode=0
Sep 09 03:31:44.073951 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.073463    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"
Sep 09 03:31:44.073951 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.073503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf" event={"ID":"c9dd2645-6ca1-46d3-9e2a-a618bad8447b","Type":"ContainerDied","Data":"cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd"}
Sep 09 03:31:44.073951 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.073559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf" event={"ID":"c9dd2645-6ca1-46d3-9e2a-a618bad8447b","Type":"ContainerDied","Data":"7c0780c9b64f610831d6ec2842971b0b487643e30a7e0875281dbf5acd78f52c"}
Sep 09 03:31:44.073951 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.073607    2580 scope.go:117] "RemoveContainer" containerID="cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd"
Sep 09 03:31:44.075048 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.075023    2580 generic.go:356] "Generic (PLEG): container finished" podID="b5eeb8b5-2f76-4e63-b503-14868ba34a26" containerID="c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d" exitCode=0
Sep 09 03:31:44.075134 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.075063    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"
Sep 09 03:31:44.075196 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.075135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w" event={"ID":"b5eeb8b5-2f76-4e63-b503-14868ba34a26","Type":"ContainerDied","Data":"c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d"}
Sep 09 03:31:44.075308 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.075293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w" event={"ID":"b5eeb8b5-2f76-4e63-b503-14868ba34a26","Type":"ContainerDied","Data":"ee117f7cc4917ea2bc6ce08af959f95b6bf0a89439875c75d82518e3c056e665"}
Sep 09 03:31:44.083344 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.083304    2580 scope.go:117] "RemoveContainer" containerID="cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd"
Sep 09 03:31:44.083676 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:31:44.083654    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd\": container with ID starting with cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd not found: ID does not exist" containerID="cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd"
Sep 09 03:31:44.083748 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.083684    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd"} err="failed to get container status \"cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd\": rpc error: code = NotFound desc = could not find container \"cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd\": container with ID starting with cb16ff8af65c6ba08e2b8799d4b498094a22a1190b52cb03f734a500a04819fd not found: ID does not exist"
Sep 09 03:31:44.083748 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.083703    2580 scope.go:117] "RemoveContainer" containerID="c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d"
Sep 09 03:31:44.091816 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.091802    2580 scope.go:117] "RemoveContainer" containerID="c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d"
Sep 09 03:31:44.092069 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:31:44.092054    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d\": container with ID starting with c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d not found: ID does not exist" containerID="c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d"
Sep 09 03:31:44.092113 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.092075    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d"} err="failed to get container status \"c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d\": rpc error: code = NotFound desc = could not find container \"c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d\": container with ID starting with c583f63f6b2ba6ffa3817590a24117aad905da4340d45e2ed191872658f31e7d not found: ID does not exist"
Sep 09 03:31:44.096154 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.096128    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"]
Sep 09 03:31:44.102096 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.102073    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec5029-dd6494774-9r4nf"]
Sep 09 03:31:44.111849 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.111828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"]
Sep 09 03:31:44.115878 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.115859    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec5029-566c486d87-tjq7w"]
Sep 09 03:31:44.971194 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.971152    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5eeb8b5-2f76-4e63-b503-14868ba34a26" path="/var/lib/kubelet/pods/b5eeb8b5-2f76-4e63-b503-14868ba34a26/volumes"
Sep 09 03:31:44.971663 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:44.971641    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9dd2645-6ca1-46d3-9e2a-a618bad8447b" path="/var/lib/kubelet/pods/c9dd2645-6ca1-46d3-9e2a-a618bad8447b/volumes"
Sep 09 03:31:49.957274 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:49.957237    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"]
Sep 09 03:31:49.957872 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:49.957853    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5eeb8b5-2f76-4e63-b503-14868ba34a26" containerName="payload-processing"
Sep 09 03:31:49.957964 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:49.957875    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5eeb8b5-2f76-4e63-b503-14868ba34a26" containerName="payload-processing"
Sep 09 03:31:49.957964 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:49.957892    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9dd2645-6ca1-46d3-9e2a-a618bad8447b" containerName="payload-pre-processing"
Sep 09 03:31:49.957964 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:49.957901    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9dd2645-6ca1-46d3-9e2a-a618bad8447b" containerName="payload-pre-processing"
Sep 09 03:31:49.958123 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:49.957993    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5eeb8b5-2f76-4e63-b503-14868ba34a26" containerName="payload-processing"
Sep 09 03:31:49.958123 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:49.958009    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9dd2645-6ca1-46d3-9e2a-a618bad8447b" containerName="payload-pre-processing"
Sep 09 03:31:49.962709 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:49.962690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"
Sep 09 03:31:49.965850 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:49.965828    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:31:49.966304 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:49.966281    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:31:49.968428 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:49.968406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"]
Sep 09 03:31:50.113334 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:50.113295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d72099-4ea3-4c14-86eb-a65f8f7eb8fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-811c96-ds74s\" (UID: \"66d72099-4ea3-4c14-86eb-a65f8f7eb8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"
Sep 09 03:31:50.213923 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:50.213836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d72099-4ea3-4c14-86eb-a65f8f7eb8fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-811c96-ds74s\" (UID: \"66d72099-4ea3-4c14-86eb-a65f8f7eb8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"
Sep 09 03:31:50.214437 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:50.214417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d72099-4ea3-4c14-86eb-a65f8f7eb8fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-811c96-ds74s\" (UID: \"66d72099-4ea3-4c14-86eb-a65f8f7eb8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"
Sep 09 03:31:50.275003 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:50.274957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"
Sep 09 03:31:50.551334 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:50.551309    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"]
Sep 09 03:31:51.115734 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:51.115639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s" event={"ID":"66d72099-4ea3-4c14-86eb-a65f8f7eb8fe","Type":"ContainerStarted","Data":"464eb423f12fdc2222a53924eb4abf5cc7e12895fddd6106839ca4de491d1067"}
Sep 09 03:31:51.115734 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:51.115694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s" event={"ID":"66d72099-4ea3-4c14-86eb-a65f8f7eb8fe","Type":"ContainerStarted","Data":"04c5ef4839702d43080bda9f369858da60a58b216560292717362f4c195fc184"}
Sep 09 03:31:51.132480 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:51.132423    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s" podStartSLOduration=2.132406951 podStartE2EDuration="2.132406951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:31:51.131096592 +0000 UTC m=+1554.657411331" watchObservedRunningTime="2026-09-09 03:31:51.132406951 +0000 UTC m=+1554.658721690"
Sep 09 03:31:52.122153 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:52.122118    2580 generic.go:356] "Generic (PLEG): container finished" podID="66d72099-4ea3-4c14-86eb-a65f8f7eb8fe" containerID="464eb423f12fdc2222a53924eb4abf5cc7e12895fddd6106839ca4de491d1067" exitCode=0
Sep 09 03:31:52.122600 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:52.122203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s" event={"ID":"66d72099-4ea3-4c14-86eb-a65f8f7eb8fe","Type":"ContainerDied","Data":"464eb423f12fdc2222a53924eb4abf5cc7e12895fddd6106839ca4de491d1067"}
Sep 09 03:31:53.261999 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:53.261967    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"
Sep 09 03:31:53.347362 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:53.347311    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d72099-4ea3-4c14-86eb-a65f8f7eb8fe-maas-api-service-ca\") pod \"66d72099-4ea3-4c14-86eb-a65f8f7eb8fe\" (UID: \"66d72099-4ea3-4c14-86eb-a65f8f7eb8fe\") "
Sep 09 03:31:53.347721 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:53.347698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d72099-4ea3-4c14-86eb-a65f8f7eb8fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66d72099-4ea3-4c14-86eb-a65f8f7eb8fe" (UID: "66d72099-4ea3-4c14-86eb-a65f8f7eb8fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:31:53.448807 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:53.448708    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d72099-4ea3-4c14-86eb-a65f8f7eb8fe-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:31:54.133807 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:54.133777    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"
Sep 09 03:31:54.133998 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:54.133817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s" event={"ID":"66d72099-4ea3-4c14-86eb-a65f8f7eb8fe","Type":"ContainerDied","Data":"04c5ef4839702d43080bda9f369858da60a58b216560292717362f4c195fc184"}
Sep 09 03:31:54.133998 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:54.133868    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04c5ef4839702d43080bda9f369858da60a58b216560292717362f4c195fc184"
Sep 09 03:31:57.098227 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:57.098190    2580 scope.go:117] "RemoveContainer" containerID="be253c297535308e359fc43f2e2abaab5f6559f9e0d07aeb18843a81a060f46a"
Sep 09 03:31:59.020230 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:31:59.020198    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:32:05.635611 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:05.635560    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"]
Sep 09 03:32:05.636112 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:05.635798    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf" podUID="d1c7d912-c0f5-40a7-8e89-f4953e060871" containerName="payload-processing" containerID="cri-o://b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854" gracePeriod=30
Sep 09 03:32:05.902899 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:05.902872    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:32:05.976919 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:05.976882    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c7d912-c0f5-40a7-8e89-f4953e060871-plugins-config-volume\") pod \"d1c7d912-c0f5-40a7-8e89-f4953e060871\" (UID: \"d1c7d912-c0f5-40a7-8e89-f4953e060871\") "
Sep 09 03:32:05.977112 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:05.976952    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfdsw\" (UniqueName: \"kubernetes.io/projected/d1c7d912-c0f5-40a7-8e89-f4953e060871-kube-api-access-tfdsw\") pod \"d1c7d912-c0f5-40a7-8e89-f4953e060871\" (UID: \"d1c7d912-c0f5-40a7-8e89-f4953e060871\") "
Sep 09 03:32:05.977321 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:05.977296    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c7d912-c0f5-40a7-8e89-f4953e060871-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1c7d912-c0f5-40a7-8e89-f4953e060871" (UID: "d1c7d912-c0f5-40a7-8e89-f4953e060871"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:05.979182 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:05.979155    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c7d912-c0f5-40a7-8e89-f4953e060871-kube-api-access-tfdsw" (OuterVolumeSpecName: "kube-api-access-tfdsw") pod "d1c7d912-c0f5-40a7-8e89-f4953e060871" (UID: "d1c7d912-c0f5-40a7-8e89-f4953e060871"). InnerVolumeSpecName "kube-api-access-tfdsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:05.996419 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:05.996383    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"]
Sep 09 03:32:06.004286 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.004263    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-811c96-ds74s"]
Sep 09 03:32:06.077756 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.077725    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c7d912-c0f5-40a7-8e89-f4953e060871-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:06.077756 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.077752    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfdsw\" (UniqueName: \"kubernetes.io/projected/d1c7d912-c0f5-40a7-8e89-f4953e060871-kube-api-access-tfdsw\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:06.195902 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.195810    2580 generic.go:356] "Generic (PLEG): container finished" podID="d1c7d912-c0f5-40a7-8e89-f4953e060871" containerID="b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854" exitCode=0
Sep 09 03:32:06.195902 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.195858    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"
Sep 09 03:32:06.195902 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.195869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf" event={"ID":"d1c7d912-c0f5-40a7-8e89-f4953e060871","Type":"ContainerDied","Data":"b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854"}
Sep 09 03:32:06.196124 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.195923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf" event={"ID":"d1c7d912-c0f5-40a7-8e89-f4953e060871","Type":"ContainerDied","Data":"c948c22cb0f5bb426edd87995451dd12cbc181b18a7a954f26b8a3fd547914de"}
Sep 09 03:32:06.196124 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.195953    2580 scope.go:117] "RemoveContainer" containerID="b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854"
Sep 09 03:32:06.212372 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.212351    2580 scope.go:117] "RemoveContainer" containerID="b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854"
Sep 09 03:32:06.212712 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:32:06.212688    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854\": container with ID starting with b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854 not found: ID does not exist" containerID="b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854"
Sep 09 03:32:06.212798 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.212720    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854"} err="failed to get container status \"b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854\": rpc error: code = NotFound desc = could not find container \"b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854\": container with ID starting with b54b037c58b4e381c28dac218f181f52702beb3af7acc2b72cedde9cce5aa854 not found: ID does not exist"
Sep 09 03:32:06.225675 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.225626    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"]
Sep 09 03:32:06.236599 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.236559    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-811c96-6cd859645-5hvxf"]
Sep 09 03:32:06.971340 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.971287    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d72099-4ea3-4c14-86eb-a65f8f7eb8fe" path="/var/lib/kubelet/pods/66d72099-4ea3-4c14-86eb-a65f8f7eb8fe/volumes"
Sep 09 03:32:06.971957 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:06.971935    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1c7d912-c0f5-40a7-8e89-f4953e060871" path="/var/lib/kubelet/pods/d1c7d912-c0f5-40a7-8e89-f4953e060871/volumes"
Sep 09 03:32:12.714080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.714046    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"]
Sep 09 03:32:12.714533 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.714451    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c7d912-c0f5-40a7-8e89-f4953e060871" containerName="payload-processing"
Sep 09 03:32:12.714533 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.714462    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c7d912-c0f5-40a7-8e89-f4953e060871" containerName="payload-processing"
Sep 09 03:32:12.714533 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.714472    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d72099-4ea3-4c14-86eb-a65f8f7eb8fe" containerName="revoke-keys"
Sep 09 03:32:12.714533 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.714478    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d72099-4ea3-4c14-86eb-a65f8f7eb8fe" containerName="revoke-keys"
Sep 09 03:32:12.714693 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.714548    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1c7d912-c0f5-40a7-8e89-f4953e060871" containerName="payload-processing"
Sep 09 03:32:12.714693 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.714558    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d72099-4ea3-4c14-86eb-a65f8f7eb8fe" containerName="revoke-keys"
Sep 09 03:32:12.718776 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.718760    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"
Sep 09 03:32:12.721099 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.721076    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:32:12.721099 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.721079    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:32:12.724766 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.724741    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"]
Sep 09 03:32:12.840033 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.840002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046afc78-2acb-4384-9c4f-0f2539675737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb\" (UID: \"046afc78-2acb-4384-9c4f-0f2539675737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"
Sep 09 03:32:12.940605 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.940554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046afc78-2acb-4384-9c4f-0f2539675737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb\" (UID: \"046afc78-2acb-4384-9c4f-0f2539675737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"
Sep 09 03:32:12.941135 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:12.941114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046afc78-2acb-4384-9c4f-0f2539675737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb\" (UID: \"046afc78-2acb-4384-9c4f-0f2539675737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"
Sep 09 03:32:13.029983 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:13.029894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"
Sep 09 03:32:13.300953 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:13.300921    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"]
Sep 09 03:32:13.303527 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:32:13.303500    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046afc78_2acb_4384_9c4f_0f2539675737.slice/crio-20126640b0521b545f46ffbfdfb3c1662be6e64b8ffed4fd038903ad65347908 WatchSource:0}: Error finding container 20126640b0521b545f46ffbfdfb3c1662be6e64b8ffed4fd038903ad65347908: Status 404 returned error can't find the container with id 20126640b0521b545f46ffbfdfb3c1662be6e64b8ffed4fd038903ad65347908
Sep 09 03:32:13.806858 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:13.806773    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:32:14.238361 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:14.238323    2580 generic.go:356] "Generic (PLEG): container finished" podID="046afc78-2acb-4384-9c4f-0f2539675737" containerID="b6d3881c45bd415493517d1c8e7ad7c0a2cae990993718b984dbb8e6cc2aeb11" exitCode=0
Sep 09 03:32:14.238532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:14.238412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb" event={"ID":"046afc78-2acb-4384-9c4f-0f2539675737","Type":"ContainerDied","Data":"b6d3881c45bd415493517d1c8e7ad7c0a2cae990993718b984dbb8e6cc2aeb11"}
Sep 09 03:32:14.238532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:14.238453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb" event={"ID":"046afc78-2acb-4384-9c4f-0f2539675737","Type":"ContainerStarted","Data":"20126640b0521b545f46ffbfdfb3c1662be6e64b8ffed4fd038903ad65347908"}
Sep 09 03:32:15.374277 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:15.374250    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"
Sep 09 03:32:15.463538 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:15.463506    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046afc78-2acb-4384-9c4f-0f2539675737-maas-api-service-ca\") pod \"046afc78-2acb-4384-9c4f-0f2539675737\" (UID: \"046afc78-2acb-4384-9c4f-0f2539675737\") "
Sep 09 03:32:15.463884 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:15.463856    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046afc78-2acb-4384-9c4f-0f2539675737-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "046afc78-2acb-4384-9c4f-0f2539675737" (UID: "046afc78-2acb-4384-9c4f-0f2539675737"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:15.565031 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:15.564948    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046afc78-2acb-4384-9c4f-0f2539675737-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:16.250367 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:16.250332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb" event={"ID":"046afc78-2acb-4384-9c4f-0f2539675737","Type":"ContainerDied","Data":"20126640b0521b545f46ffbfdfb3c1662be6e64b8ffed4fd038903ad65347908"}
Sep 09 03:32:16.250555 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:16.250388    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20126640b0521b545f46ffbfdfb3c1662be6e64b8ffed4fd038903ad65347908"
Sep 09 03:32:16.250555 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:16.250343    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"
Sep 09 03:32:23.538128 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.538085    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"]
Sep 09 03:32:23.538636 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.538324    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p" podUID="63947e97-2e0e-49df-8805-b3ec7a626f04" containerName="payload-processing" containerID="cri-o://13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560" gracePeriod=30
Sep 09 03:32:23.589411 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.589375    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"]
Sep 09 03:32:23.589656 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.589632    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn" podUID="860317db-ee8c-4f93-863d-7ac3ead55cc1" containerName="payload-pre-processing" containerID="cri-o://2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae" gracePeriod=30
Sep 09 03:32:23.872426 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.872400    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:32:23.875382 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.875366    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:32:23.942040 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.942011    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63947e97-2e0e-49df-8805-b3ec7a626f04-plugins-config-volume\") pod \"63947e97-2e0e-49df-8805-b3ec7a626f04\" (UID: \"63947e97-2e0e-49df-8805-b3ec7a626f04\") "
Sep 09 03:32:23.942040 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.942047    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdprl\" (UniqueName: \"kubernetes.io/projected/860317db-ee8c-4f93-863d-7ac3ead55cc1-kube-api-access-hdprl\") pod \"860317db-ee8c-4f93-863d-7ac3ead55cc1\" (UID: \"860317db-ee8c-4f93-863d-7ac3ead55cc1\") "
Sep 09 03:32:23.942296 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.942065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860317db-ee8c-4f93-863d-7ac3ead55cc1-plugins-config-volume\") pod \"860317db-ee8c-4f93-863d-7ac3ead55cc1\" (UID: \"860317db-ee8c-4f93-863d-7ac3ead55cc1\") "
Sep 09 03:32:23.942296 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.942108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjb7b\" (UniqueName: \"kubernetes.io/projected/63947e97-2e0e-49df-8805-b3ec7a626f04-kube-api-access-cjb7b\") pod \"63947e97-2e0e-49df-8805-b3ec7a626f04\" (UID: \"63947e97-2e0e-49df-8805-b3ec7a626f04\") "
Sep 09 03:32:23.942474 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.942436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860317db-ee8c-4f93-863d-7ac3ead55cc1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "860317db-ee8c-4f93-863d-7ac3ead55cc1" (UID: "860317db-ee8c-4f93-863d-7ac3ead55cc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:23.942603 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.942476    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63947e97-2e0e-49df-8805-b3ec7a626f04-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63947e97-2e0e-49df-8805-b3ec7a626f04" (UID: "63947e97-2e0e-49df-8805-b3ec7a626f04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:23.944393 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.944359    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860317db-ee8c-4f93-863d-7ac3ead55cc1-kube-api-access-hdprl" (OuterVolumeSpecName: "kube-api-access-hdprl") pod "860317db-ee8c-4f93-863d-7ac3ead55cc1" (UID: "860317db-ee8c-4f93-863d-7ac3ead55cc1"). InnerVolumeSpecName "kube-api-access-hdprl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:23.944507 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.944418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63947e97-2e0e-49df-8805-b3ec7a626f04-kube-api-access-cjb7b" (OuterVolumeSpecName: "kube-api-access-cjb7b") pod "63947e97-2e0e-49df-8805-b3ec7a626f04" (UID: "63947e97-2e0e-49df-8805-b3ec7a626f04"). InnerVolumeSpecName "kube-api-access-cjb7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:23.966238 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.966210    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"]
Sep 09 03:32:23.971378 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:23.971350    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-207ebd-9cfzb"]
Sep 09 03:32:24.043474 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.043436    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjb7b\" (UniqueName: \"kubernetes.io/projected/63947e97-2e0e-49df-8805-b3ec7a626f04-kube-api-access-cjb7b\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:24.043474 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.043465    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63947e97-2e0e-49df-8805-b3ec7a626f04-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:24.043474 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.043474    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdprl\" (UniqueName: \"kubernetes.io/projected/860317db-ee8c-4f93-863d-7ac3ead55cc1-kube-api-access-hdprl\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:24.043474 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.043482    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860317db-ee8c-4f93-863d-7ac3ead55cc1-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:24.290197 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.290158    2580 generic.go:356] "Generic (PLEG): container finished" podID="860317db-ee8c-4f93-863d-7ac3ead55cc1" containerID="2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae" exitCode=0
Sep 09 03:32:24.290380 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.290205    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"
Sep 09 03:32:24.290380 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.290254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn" event={"ID":"860317db-ee8c-4f93-863d-7ac3ead55cc1","Type":"ContainerDied","Data":"2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae"}
Sep 09 03:32:24.290380 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.290307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn" event={"ID":"860317db-ee8c-4f93-863d-7ac3ead55cc1","Type":"ContainerDied","Data":"ff68f06e3e1fde4abbb38983c52d1a2381b5c735eef2915da0bf8b6e699b04f6"}
Sep 09 03:32:24.290380 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.290336    2580 scope.go:117] "RemoveContainer" containerID="2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae"
Sep 09 03:32:24.291724 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.291703    2580 generic.go:356] "Generic (PLEG): container finished" podID="63947e97-2e0e-49df-8805-b3ec7a626f04" containerID="13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560" exitCode=0
Sep 09 03:32:24.291821 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.291745    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"
Sep 09 03:32:24.291821 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.291775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p" event={"ID":"63947e97-2e0e-49df-8805-b3ec7a626f04","Type":"ContainerDied","Data":"13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560"}
Sep 09 03:32:24.291821 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.291805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p" event={"ID":"63947e97-2e0e-49df-8805-b3ec7a626f04","Type":"ContainerDied","Data":"b6938b8a6656e42e10f1a2313968fdf86705bad6e55fb47f968c9ae73c199655"}
Sep 09 03:32:24.300445 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.300430    2580 scope.go:117] "RemoveContainer" containerID="2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae"
Sep 09 03:32:24.300811 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:32:24.300792    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae\": container with ID starting with 2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae not found: ID does not exist" containerID="2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae"
Sep 09 03:32:24.300905 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.300817    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae"} err="failed to get container status \"2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae\": rpc error: code = NotFound desc = could not find container \"2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae\": container with ID starting with 2f5e0b15415b4b9e3e57e753825705ecf7a7a8790bcbe257d851136d43d966ae not found: ID does not exist"
Sep 09 03:32:24.300905 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.300834    2580 scope.go:117] "RemoveContainer" containerID="13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560"
Sep 09 03:32:24.308920 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.308902    2580 scope.go:117] "RemoveContainer" containerID="13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560"
Sep 09 03:32:24.309176 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:32:24.309158    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560\": container with ID starting with 13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560 not found: ID does not exist" containerID="13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560"
Sep 09 03:32:24.309223 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.309184    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560"} err="failed to get container status \"13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560\": rpc error: code = NotFound desc = could not find container \"13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560\": container with ID starting with 13302c532337ee80a3a37673b0958570c3bfe0a00d9c4d14e052ac78730be560 not found: ID does not exist"
Sep 09 03:32:24.313062 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.313039    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"]
Sep 09 03:32:24.318251 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.318232    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-207ebd-7d7d646f74-llzxn"]
Sep 09 03:32:24.330470 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.330447    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"]
Sep 09 03:32:24.332325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.332305    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-207ebd-cfb8f5c6f-62v5p"]
Sep 09 03:32:24.970766 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.970734    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046afc78-2acb-4384-9c4f-0f2539675737" path="/var/lib/kubelet/pods/046afc78-2acb-4384-9c4f-0f2539675737/volumes"
Sep 09 03:32:24.971158 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.971111    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63947e97-2e0e-49df-8805-b3ec7a626f04" path="/var/lib/kubelet/pods/63947e97-2e0e-49df-8805-b3ec7a626f04/volumes"
Sep 09 03:32:24.971419 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:24.971406    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="860317db-ee8c-4f93-863d-7ac3ead55cc1" path="/var/lib/kubelet/pods/860317db-ee8c-4f93-863d-7ac3ead55cc1/volumes"
Sep 09 03:32:30.431340 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:30.431301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:32:32.293864 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.293824    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"]
Sep 09 03:32:32.294291 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.294203    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63947e97-2e0e-49df-8805-b3ec7a626f04" containerName="payload-processing"
Sep 09 03:32:32.294291 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.294215    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63947e97-2e0e-49df-8805-b3ec7a626f04" containerName="payload-processing"
Sep 09 03:32:32.294291 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.294241    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046afc78-2acb-4384-9c4f-0f2539675737" containerName="revoke-keys"
Sep 09 03:32:32.294291 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.294246    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="046afc78-2acb-4384-9c4f-0f2539675737" containerName="revoke-keys"
Sep 09 03:32:32.294291 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.294254    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="860317db-ee8c-4f93-863d-7ac3ead55cc1" containerName="payload-pre-processing"
Sep 09 03:32:32.294291 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.294260    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="860317db-ee8c-4f93-863d-7ac3ead55cc1" containerName="payload-pre-processing"
Sep 09 03:32:32.294511 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.294315    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="046afc78-2acb-4384-9c4f-0f2539675737" containerName="revoke-keys"
Sep 09 03:32:32.294511 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.294326    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="860317db-ee8c-4f93-863d-7ac3ead55cc1" containerName="payload-pre-processing"
Sep 09 03:32:32.294511 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.294333    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="63947e97-2e0e-49df-8805-b3ec7a626f04" containerName="payload-processing"
Sep 09 03:32:32.297349 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.297332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"
Sep 09 03:32:32.303418 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.303393    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:32:32.303835 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.303816    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:32:32.315986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.315960    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"]
Sep 09 03:32:32.422824 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.422789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44a43a0-6d39-4164-916c-77663caae905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z\" (UID: \"e44a43a0-6d39-4164-916c-77663caae905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"
Sep 09 03:32:32.523536 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.523498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44a43a0-6d39-4164-916c-77663caae905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z\" (UID: \"e44a43a0-6d39-4164-916c-77663caae905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"
Sep 09 03:32:32.524124 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.524105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44a43a0-6d39-4164-916c-77663caae905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z\" (UID: \"e44a43a0-6d39-4164-916c-77663caae905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"
Sep 09 03:32:32.607545 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.607511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"
Sep 09 03:32:32.887594 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:32.887546    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"]
Sep 09 03:32:32.888950 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:32:32.888925    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44a43a0_6d39_4164_916c_77663caae905.slice/crio-1b0a6a1e801b7be37626003b63ac5a69f672e61435ddc26fcf498f1ea59c2c6c WatchSource:0}: Error finding container 1b0a6a1e801b7be37626003b63ac5a69f672e61435ddc26fcf498f1ea59c2c6c: Status 404 returned error can't find the container with id 1b0a6a1e801b7be37626003b63ac5a69f672e61435ddc26fcf498f1ea59c2c6c
Sep 09 03:32:33.344570 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:33.344538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z" event={"ID":"e44a43a0-6d39-4164-916c-77663caae905","Type":"ContainerStarted","Data":"275ddb4e788662a8022b29a30941ac068d98f57f1375c82ca5cb3a4331071740"}
Sep 09 03:32:33.345045 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:33.344601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z" event={"ID":"e44a43a0-6d39-4164-916c-77663caae905","Type":"ContainerStarted","Data":"1b0a6a1e801b7be37626003b63ac5a69f672e61435ddc26fcf498f1ea59c2c6c"}
Sep 09 03:32:33.368023 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:33.367975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z" podStartSLOduration=1.367961595 podStartE2EDuration="1.367961595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:32:33.367611405 +0000 UTC m=+1596.893926141" watchObservedRunningTime="2026-09-09 03:32:33.367961595 +0000 UTC m=+1596.894276341"
Sep 09 03:32:34.357485 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:34.357449    2580 generic.go:356] "Generic (PLEG): container finished" podID="e44a43a0-6d39-4164-916c-77663caae905" containerID="275ddb4e788662a8022b29a30941ac068d98f57f1375c82ca5cb3a4331071740" exitCode=0
Sep 09 03:32:34.357881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:34.357534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z" event={"ID":"e44a43a0-6d39-4164-916c-77663caae905","Type":"ContainerDied","Data":"275ddb4e788662a8022b29a30941ac068d98f57f1375c82ca5cb3a4331071740"}
Sep 09 03:32:35.500646 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:35.500621    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"
Sep 09 03:32:35.654139 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:35.654044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44a43a0-6d39-4164-916c-77663caae905-maas-api-service-ca\") pod \"e44a43a0-6d39-4164-916c-77663caae905\" (UID: \"e44a43a0-6d39-4164-916c-77663caae905\") "
Sep 09 03:32:35.654423 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:35.654403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44a43a0-6d39-4164-916c-77663caae905-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e44a43a0-6d39-4164-916c-77663caae905" (UID: "e44a43a0-6d39-4164-916c-77663caae905"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:35.754991 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:35.754954    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44a43a0-6d39-4164-916c-77663caae905-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:36.369214 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:36.369175    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"
Sep 09 03:32:36.369383 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:36.369174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z" event={"ID":"e44a43a0-6d39-4164-916c-77663caae905","Type":"ContainerDied","Data":"1b0a6a1e801b7be37626003b63ac5a69f672e61435ddc26fcf498f1ea59c2c6c"}
Sep 09 03:32:36.369383 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:36.369316    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b0a6a1e801b7be37626003b63ac5a69f672e61435ddc26fcf498f1ea59c2c6c"
Sep 09 03:32:43.207972 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.207935    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"]
Sep 09 03:32:43.208415 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.208175    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t" podUID="77c3af93-5b53-4cb5-a0e8-9ba746113e21" containerName="payload-processing" containerID="cri-o://146cec505056f82fef015b70e5d834a840b8bb7f8a349576b57c8e97b9afe2e4" gracePeriod=30
Sep 09 03:32:43.233086 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.233056    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"]
Sep 09 03:32:43.233361 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.233315    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t" podUID="50f78d04-56ed-4bc8-8df8-f18a243d4b00" containerName="payload-pre-processing" containerID="cri-o://e962d4492cbc54c26d4fde990ae003692f961a71aa8e0098dec22cc575c7b498" gracePeriod=30
Sep 09 03:32:43.409007 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.408967    2580 generic.go:356] "Generic (PLEG): container finished" podID="77c3af93-5b53-4cb5-a0e8-9ba746113e21" containerID="146cec505056f82fef015b70e5d834a840b8bb7f8a349576b57c8e97b9afe2e4" exitCode=0
Sep 09 03:32:43.409159 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.409105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t" event={"ID":"77c3af93-5b53-4cb5-a0e8-9ba746113e21","Type":"ContainerDied","Data":"146cec505056f82fef015b70e5d834a840b8bb7f8a349576b57c8e97b9afe2e4"}
Sep 09 03:32:43.410778 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.410753    2580 generic.go:356] "Generic (PLEG): container finished" podID="50f78d04-56ed-4bc8-8df8-f18a243d4b00" containerID="e962d4492cbc54c26d4fde990ae003692f961a71aa8e0098dec22cc575c7b498" exitCode=0
Sep 09 03:32:43.410861 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.410819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t" event={"ID":"50f78d04-56ed-4bc8-8df8-f18a243d4b00","Type":"ContainerDied","Data":"e962d4492cbc54c26d4fde990ae003692f961a71aa8e0098dec22cc575c7b498"}
Sep 09 03:32:43.509291 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.509267    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:32:43.512296 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.512270    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:32:43.526958 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.526937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c3af93-5b53-4cb5-a0e8-9ba746113e21-plugins-config-volume\") pod \"77c3af93-5b53-4cb5-a0e8-9ba746113e21\" (UID: \"77c3af93-5b53-4cb5-a0e8-9ba746113e21\") "
Sep 09 03:32:43.527053 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.526978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m657f\" (UniqueName: \"kubernetes.io/projected/50f78d04-56ed-4bc8-8df8-f18a243d4b00-kube-api-access-m657f\") pod \"50f78d04-56ed-4bc8-8df8-f18a243d4b00\" (UID: \"50f78d04-56ed-4bc8-8df8-f18a243d4b00\") "
Sep 09 03:32:43.527189 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.527171    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5zf5\" (UniqueName: \"kubernetes.io/projected/77c3af93-5b53-4cb5-a0e8-9ba746113e21-kube-api-access-n5zf5\") pod \"77c3af93-5b53-4cb5-a0e8-9ba746113e21\" (UID: \"77c3af93-5b53-4cb5-a0e8-9ba746113e21\") "
Sep 09 03:32:43.527265 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.527215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f78d04-56ed-4bc8-8df8-f18a243d4b00-plugins-config-volume\") pod \"50f78d04-56ed-4bc8-8df8-f18a243d4b00\" (UID: \"50f78d04-56ed-4bc8-8df8-f18a243d4b00\") "
Sep 09 03:32:43.527347 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.527327    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c3af93-5b53-4cb5-a0e8-9ba746113e21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77c3af93-5b53-4cb5-a0e8-9ba746113e21" (UID: "77c3af93-5b53-4cb5-a0e8-9ba746113e21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:43.527505 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.527482    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c3af93-5b53-4cb5-a0e8-9ba746113e21-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:43.527656 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.527619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f78d04-56ed-4bc8-8df8-f18a243d4b00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50f78d04-56ed-4bc8-8df8-f18a243d4b00" (UID: "50f78d04-56ed-4bc8-8df8-f18a243d4b00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:43.529318 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.529285    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f78d04-56ed-4bc8-8df8-f18a243d4b00-kube-api-access-m657f" (OuterVolumeSpecName: "kube-api-access-m657f") pod "50f78d04-56ed-4bc8-8df8-f18a243d4b00" (UID: "50f78d04-56ed-4bc8-8df8-f18a243d4b00"). InnerVolumeSpecName "kube-api-access-m657f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:43.529411 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.529375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c3af93-5b53-4cb5-a0e8-9ba746113e21-kube-api-access-n5zf5" (OuterVolumeSpecName: "kube-api-access-n5zf5") pod "77c3af93-5b53-4cb5-a0e8-9ba746113e21" (UID: "77c3af93-5b53-4cb5-a0e8-9ba746113e21"). InnerVolumeSpecName "kube-api-access-n5zf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:43.539188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.539168    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"]
Sep 09 03:32:43.555856 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.555832    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4c8420-2966z"]
Sep 09 03:32:43.628134 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.628104    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5zf5\" (UniqueName: \"kubernetes.io/projected/77c3af93-5b53-4cb5-a0e8-9ba746113e21-kube-api-access-n5zf5\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:43.628134 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.628136    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f78d04-56ed-4bc8-8df8-f18a243d4b00-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:43.628347 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:43.628152    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m657f\" (UniqueName: \"kubernetes.io/projected/50f78d04-56ed-4bc8-8df8-f18a243d4b00-kube-api-access-m657f\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:32:44.417682 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.417646    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"
Sep 09 03:32:44.417682 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.417676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t" event={"ID":"50f78d04-56ed-4bc8-8df8-f18a243d4b00","Type":"ContainerDied","Data":"d8a6d20005c6912a25ea127f9da37ef7262902f4658be8690bfd5a2dfa12ed87"}
Sep 09 03:32:44.418312 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.417733    2580 scope.go:117] "RemoveContainer" containerID="e962d4492cbc54c26d4fde990ae003692f961a71aa8e0098dec22cc575c7b498"
Sep 09 03:32:44.419288 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.419269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t" event={"ID":"77c3af93-5b53-4cb5-a0e8-9ba746113e21","Type":"ContainerDied","Data":"5b952d68014cf8f192f5bcee4ed07b94972b0e87000f6a34f1c4ba620846a135"}
Sep 09 03:32:44.419288 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.419287    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"
Sep 09 03:32:44.427532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.427512    2580 scope.go:117] "RemoveContainer" containerID="146cec505056f82fef015b70e5d834a840b8bb7f8a349576b57c8e97b9afe2e4"
Sep 09 03:32:44.444921 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.444898    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"]
Sep 09 03:32:44.453948 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.453924    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4c8420-6bd9c5bcbf-w895t"]
Sep 09 03:32:44.474555 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.474528    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"]
Sep 09 03:32:44.483027 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.482999    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4c8420-6c6669b96b-9xr5t"]
Sep 09 03:32:44.971317 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.971277    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f78d04-56ed-4bc8-8df8-f18a243d4b00" path="/var/lib/kubelet/pods/50f78d04-56ed-4bc8-8df8-f18a243d4b00/volumes"
Sep 09 03:32:44.971759 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.971738    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77c3af93-5b53-4cb5-a0e8-9ba746113e21" path="/var/lib/kubelet/pods/77c3af93-5b53-4cb5-a0e8-9ba746113e21/volumes"
Sep 09 03:32:44.972094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:44.972077    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e44a43a0-6d39-4164-916c-77663caae905" path="/var/lib/kubelet/pods/e44a43a0-6d39-4164-916c-77663caae905/volumes"
Sep 09 03:32:46.961088 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:46.961052    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:32:57.609343 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:32:57.609301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:33:13.997380 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:13.997345    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:33:23.511612 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:23.511556    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:33:34.357565 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:34.357530    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:33:37.331615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.331565    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"]
Sep 09 03:33:37.331983 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.331941    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44a43a0-6d39-4164-916c-77663caae905" containerName="revoke-keys"
Sep 09 03:33:37.331983 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.331952    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44a43a0-6d39-4164-916c-77663caae905" containerName="revoke-keys"
Sep 09 03:33:37.331983 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.331965    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f78d04-56ed-4bc8-8df8-f18a243d4b00" containerName="payload-pre-processing"
Sep 09 03:33:37.331983 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.331970    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f78d04-56ed-4bc8-8df8-f18a243d4b00" containerName="payload-pre-processing"
Sep 09 03:33:37.331983 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.331977    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c3af93-5b53-4cb5-a0e8-9ba746113e21" containerName="payload-processing"
Sep 09 03:33:37.331983 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.331983    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c3af93-5b53-4cb5-a0e8-9ba746113e21" containerName="payload-processing"
Sep 09 03:33:37.332186 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.332058    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="77c3af93-5b53-4cb5-a0e8-9ba746113e21" containerName="payload-processing"
Sep 09 03:33:37.332186 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.332068    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f78d04-56ed-4bc8-8df8-f18a243d4b00" containerName="payload-pre-processing"
Sep 09 03:33:37.332186 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.332074    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44a43a0-6d39-4164-916c-77663caae905" containerName="revoke-keys"
Sep 09 03:33:37.335186 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.335167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:33:37.339793 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.339773    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ed81bb\""
Sep 09 03:33:37.339903 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.339822    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ed81bb-dockercfg-gjdpg\""
Sep 09 03:33:37.358562 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.358526    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"]
Sep 09 03:33:37.374088 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.374053    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"]
Sep 09 03:33:37.377894 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.377873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:33:37.393004 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.392979    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"]
Sep 09 03:33:37.406749 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.406715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12cbae4-cbb2-4292-8617-b908fa3d308e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm\" (UID: \"f12cbae4-cbb2-4292-8617-b908fa3d308e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:33:37.406877 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.406755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7v7x\" (UniqueName: \"kubernetes.io/projected/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-kube-api-access-k7v7x\") pod \"payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c\" (UID: \"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:33:37.406877 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.406786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktrw\" (UniqueName: \"kubernetes.io/projected/f12cbae4-cbb2-4292-8617-b908fa3d308e-kube-api-access-sktrw\") pod \"payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm\" (UID: \"f12cbae4-cbb2-4292-8617-b908fa3d308e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:33:37.406877 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.406846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c\" (UID: \"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:33:37.507682 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.507646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12cbae4-cbb2-4292-8617-b908fa3d308e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm\" (UID: \"f12cbae4-cbb2-4292-8617-b908fa3d308e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:33:37.507682 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.507689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7v7x\" (UniqueName: \"kubernetes.io/projected/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-kube-api-access-k7v7x\") pod \"payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c\" (UID: \"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:33:37.507947 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.507712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktrw\" (UniqueName: \"kubernetes.io/projected/f12cbae4-cbb2-4292-8617-b908fa3d308e-kube-api-access-sktrw\") pod \"payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm\" (UID: \"f12cbae4-cbb2-4292-8617-b908fa3d308e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:33:37.507947 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.507738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c\" (UID: \"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:33:37.508470 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.508444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c\" (UID: \"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:33:37.508516 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.508444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12cbae4-cbb2-4292-8617-b908fa3d308e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm\" (UID: \"f12cbae4-cbb2-4292-8617-b908fa3d308e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:33:37.519930 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.519902    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7v7x\" (UniqueName: \"kubernetes.io/projected/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-kube-api-access-k7v7x\") pod \"payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c\" (UID: \"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:33:37.521491 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.521474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktrw\" (UniqueName: \"kubernetes.io/projected/f12cbae4-cbb2-4292-8617-b908fa3d308e-kube-api-access-sktrw\") pod \"payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm\" (UID: \"f12cbae4-cbb2-4292-8617-b908fa3d308e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:33:37.645757 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.645717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:33:37.690346 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.690319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:33:37.924789 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.924749    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"]
Sep 09 03:33:37.926967 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:33:37.926944    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f41d8ba_03b7_4409_bd3b_ec1b782e28fc.slice/crio-7ece42746a877b0dac9711327ad014db4f77698a842801ace6e6af67f58632c3 WatchSource:0}: Error finding container 7ece42746a877b0dac9711327ad014db4f77698a842801ace6e6af67f58632c3: Status 404 returned error can't find the container with id 7ece42746a877b0dac9711327ad014db4f77698a842801ace6e6af67f58632c3
Sep 09 03:33:37.976212 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:37.976158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"]
Sep 09 03:33:37.979027 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:33:37.978996    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12cbae4_cbb2_4292_8617_b908fa3d308e.slice/crio-5f8be772b5dc161444eeba77d0d13d638d341ffe44448ef2d64388d3d353b16d WatchSource:0}: Error finding container 5f8be772b5dc161444eeba77d0d13d638d341ffe44448ef2d64388d3d353b16d: Status 404 returned error can't find the container with id 5f8be772b5dc161444eeba77d0d13d638d341ffe44448ef2d64388d3d353b16d
Sep 09 03:33:38.696759 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:38.696726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm" event={"ID":"f12cbae4-cbb2-4292-8617-b908fa3d308e","Type":"ContainerStarted","Data":"26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e"}
Sep 09 03:33:38.697259 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:38.696775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm" event={"ID":"f12cbae4-cbb2-4292-8617-b908fa3d308e","Type":"ContainerStarted","Data":"5f8be772b5dc161444eeba77d0d13d638d341ffe44448ef2d64388d3d353b16d"}
Sep 09 03:33:38.697259 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:38.696843    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:33:38.698563 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:38.698538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c" event={"ID":"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc","Type":"ContainerStarted","Data":"5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89"}
Sep 09 03:33:38.698652 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:38.698598    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c" event={"ID":"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc","Type":"ContainerStarted","Data":"7ece42746a877b0dac9711327ad014db4f77698a842801ace6e6af67f58632c3"}
Sep 09 03:33:38.698652 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:38.698624    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:33:38.734621 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:38.734545    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm" podStartSLOduration=1.734528163 podStartE2EDuration="1.734528163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:33:38.733301587 +0000 UTC m=+1662.259616327" watchObservedRunningTime="2026-09-09 03:33:38.734528163 +0000 UTC m=+1662.260842902"
Sep 09 03:33:38.773133 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:38.773078    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c" podStartSLOduration=1.773060461 podStartE2EDuration="1.773060461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:33:38.770971116 +0000 UTC m=+1662.297285856" watchObservedRunningTime="2026-09-09 03:33:38.773060461 +0000 UTC m=+1662.299375196"
Sep 09 03:33:46.856778 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:46.856745    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:33:49.705325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:49.705290    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:33:49.705795 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:33:49.705376    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:34:12.427806 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:34:12.427770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:34:35.097043 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:34:35.096975    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kmv79_dcbb770c-adde-492a-92b4-22a3050ced12/payload-processing/0.log"
Sep 09 03:34:44.305058 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:34:44.305024    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:34:59.511676 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:34:59.511634    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:35:05.855484 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:05.855452    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm_f12cbae4-cbb2-4292-8617-b908fa3d308e/payload-processing/0.log"
Sep 09 03:35:05.990585 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:05.990557    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kmv79_dcbb770c-adde-492a-92b4-22a3050ced12/payload-processing/0.log"
Sep 09 03:35:11.714059 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.714020    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"]
Sep 09 03:35:11.717741 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.717724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:11.720809 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.720788    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-7e9fa8-dockercfg-rdn4q\""
Sep 09 03:35:11.720929 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.720796    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-7e9fa8\""
Sep 09 03:35:11.728625 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.728568    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"]
Sep 09 03:35:11.741681 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.741641    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"]
Sep 09 03:35:11.746543 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.746519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:11.757531 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.757504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"]
Sep 09 03:35:11.896450 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.896411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k\" (UID: \"1044bb12-9ca3-4a04-90d7-3a5e23e3274c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:11.896653 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.896459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5hsn\" (UniqueName: \"kubernetes.io/projected/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-kube-api-access-v5hsn\") pod \"payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k\" (UID: \"1044bb12-9ca3-4a04-90d7-3a5e23e3274c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:11.896653 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.896526    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53365a77-484a-4820-9477-0fb85f0e70c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf\" (UID: \"53365a77-484a-4820-9477-0fb85f0e70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:11.896653 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.896553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbmdc\" (UniqueName: \"kubernetes.io/projected/53365a77-484a-4820-9477-0fb85f0e70c0-kube-api-access-bbmdc\") pod \"payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf\" (UID: \"53365a77-484a-4820-9477-0fb85f0e70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:11.998129 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.998029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmdc\" (UniqueName: \"kubernetes.io/projected/53365a77-484a-4820-9477-0fb85f0e70c0-kube-api-access-bbmdc\") pod \"payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf\" (UID: \"53365a77-484a-4820-9477-0fb85f0e70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:11.998129 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.998099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k\" (UID: \"1044bb12-9ca3-4a04-90d7-3a5e23e3274c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:11.998129 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.998125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5hsn\" (UniqueName: \"kubernetes.io/projected/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-kube-api-access-v5hsn\") pod \"payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k\" (UID: \"1044bb12-9ca3-4a04-90d7-3a5e23e3274c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:11.998532 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.998168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53365a77-484a-4820-9477-0fb85f0e70c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf\" (UID: \"53365a77-484a-4820-9477-0fb85f0e70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:11.998972 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.998947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53365a77-484a-4820-9477-0fb85f0e70c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf\" (UID: \"53365a77-484a-4820-9477-0fb85f0e70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:11.999064 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:11.998999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k\" (UID: \"1044bb12-9ca3-4a04-90d7-3a5e23e3274c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:12.007115 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:12.007092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmdc\" (UniqueName: \"kubernetes.io/projected/53365a77-484a-4820-9477-0fb85f0e70c0-kube-api-access-bbmdc\") pod \"payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf\" (UID: \"53365a77-484a-4820-9477-0fb85f0e70c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:12.007263 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:12.007242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5hsn\" (UniqueName: \"kubernetes.io/projected/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-kube-api-access-v5hsn\") pod \"payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k\" (UID: \"1044bb12-9ca3-4a04-90d7-3a5e23e3274c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:12.030083 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:12.030060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:12.059134 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:12.059093    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:12.320070 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:12.320046    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"]
Sep 09 03:35:12.365472 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:12.365447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"]
Sep 09 03:35:12.366048 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:35:12.366027    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1044bb12_9ca3_4a04_90d7_3a5e23e3274c.slice/crio-62c3056c6fe9b7f39f25af039ba8c1680fc2c83428437cf13e8267cad78ef12d WatchSource:0}: Error finding container 62c3056c6fe9b7f39f25af039ba8c1680fc2c83428437cf13e8267cad78ef12d: Status 404 returned error can't find the container with id 62c3056c6fe9b7f39f25af039ba8c1680fc2c83428437cf13e8267cad78ef12d
Sep 09 03:35:13.180083 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:13.180042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf" event={"ID":"53365a77-484a-4820-9477-0fb85f0e70c0","Type":"ContainerStarted","Data":"4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331"}
Sep 09 03:35:13.180547 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:13.180103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf" event={"ID":"53365a77-484a-4820-9477-0fb85f0e70c0","Type":"ContainerStarted","Data":"c028867f6fb54670bf5a129c84a9411bf3ac076162435c4a18948643ba2eac7b"}
Sep 09 03:35:13.180547 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:13.180134    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:13.181712 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:13.181685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k" event={"ID":"1044bb12-9ca3-4a04-90d7-3a5e23e3274c","Type":"ContainerStarted","Data":"772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226"}
Sep 09 03:35:13.181831 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:13.181731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k" event={"ID":"1044bb12-9ca3-4a04-90d7-3a5e23e3274c","Type":"ContainerStarted","Data":"62c3056c6fe9b7f39f25af039ba8c1680fc2c83428437cf13e8267cad78ef12d"}
Sep 09 03:35:13.181831 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:13.181750    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:13.197739 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:13.197695    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf" podStartSLOduration=2.197679972 podStartE2EDuration="2.197679972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:35:13.196223797 +0000 UTC m=+1756.722538547" watchObservedRunningTime="2026-09-09 03:35:13.197679972 +0000 UTC m=+1756.723994721"
Sep 09 03:35:13.213733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:13.213685    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k" podStartSLOduration=2.213669709 podStartE2EDuration="2.213669709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:35:13.211787691 +0000 UTC m=+1756.738102431" watchObservedRunningTime="2026-09-09 03:35:13.213669709 +0000 UTC m=+1756.739984448"
Sep 09 03:35:19.817368 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:19.817327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:35:24.189243 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:24.189209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:24.189655 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:24.189287    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:35.627910 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:35.627874    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"]
Sep 09 03:35:35.631341 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:35.631325    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"
Sep 09 03:35:35.633845 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:35.633815    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:35:35.634841 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:35.634824    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:35:35.640929 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:35.640905    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"]
Sep 09 03:35:35.726498 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:35.726455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c95f54-7c9d-4a97-9f43-bf844278e765-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf\" (UID: \"a1c95f54-7c9d-4a97-9f43-bf844278e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"
Sep 09 03:35:35.828012 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:35.827981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c95f54-7c9d-4a97-9f43-bf844278e765-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf\" (UID: \"a1c95f54-7c9d-4a97-9f43-bf844278e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"
Sep 09 03:35:35.828555 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:35.828537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c95f54-7c9d-4a97-9f43-bf844278e765-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf\" (UID: \"a1c95f54-7c9d-4a97-9f43-bf844278e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"
Sep 09 03:35:35.943672 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:35.943558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"
Sep 09 03:35:36.215677 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:36.215652    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"]
Sep 09 03:35:36.217765 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:35:36.217736    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c95f54_7c9d_4a97_9f43_bf844278e765.slice/crio-d28c792052171545826430751164fdfd089b5e4ceb6b63ea98adb3d97ee1882c WatchSource:0}: Error finding container d28c792052171545826430751164fdfd089b5e4ceb6b63ea98adb3d97ee1882c: Status 404 returned error can't find the container with id d28c792052171545826430751164fdfd089b5e4ceb6b63ea98adb3d97ee1882c
Sep 09 03:35:36.301349 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:36.301318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf" event={"ID":"a1c95f54-7c9d-4a97-9f43-bf844278e765","Type":"ContainerStarted","Data":"d28c792052171545826430751164fdfd089b5e4ceb6b63ea98adb3d97ee1882c"}
Sep 09 03:35:37.308845 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:37.308767    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1c95f54-7c9d-4a97-9f43-bf844278e765" containerID="247193df10cb753044be457ff3b0e8cdd8cb6281e19c80866125161d3da11e49" exitCode=0
Sep 09 03:35:37.308845 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:37.308820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf" event={"ID":"a1c95f54-7c9d-4a97-9f43-bf844278e765","Type":"ContainerDied","Data":"247193df10cb753044be457ff3b0e8cdd8cb6281e19c80866125161d3da11e49"}
Sep 09 03:35:38.453938 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:38.453915    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"
Sep 09 03:35:38.556007 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:38.555972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c95f54-7c9d-4a97-9f43-bf844278e765-maas-api-service-ca\") pod \"a1c95f54-7c9d-4a97-9f43-bf844278e765\" (UID: \"a1c95f54-7c9d-4a97-9f43-bf844278e765\") "
Sep 09 03:35:38.556370 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:38.556341    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c95f54-7c9d-4a97-9f43-bf844278e765-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1c95f54-7c9d-4a97-9f43-bf844278e765" (UID: "a1c95f54-7c9d-4a97-9f43-bf844278e765"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:35:38.656953 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:38.656921    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c95f54-7c9d-4a97-9f43-bf844278e765-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:35:39.320739 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:39.320649    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"
Sep 09 03:35:39.320739 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:39.320688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf" event={"ID":"a1c95f54-7c9d-4a97-9f43-bf844278e765","Type":"ContainerDied","Data":"d28c792052171545826430751164fdfd089b5e4ceb6b63ea98adb3d97ee1882c"}
Sep 09 03:35:39.320950 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:39.320750    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d28c792052171545826430751164fdfd089b5e4ceb6b63ea98adb3d97ee1882c"
Sep 09 03:35:41.762277 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:41.762237    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"]
Sep 09 03:35:41.763220 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:41.762782    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k" podUID="1044bb12-9ca3-4a04-90d7-3a5e23e3274c" containerName="payload-processing" containerID="cri-o://772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226" gracePeriod=30
Sep 09 03:35:41.769554 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:41.769526    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"]
Sep 09 03:35:41.769796 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:41.769771    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf" podUID="53365a77-484a-4820-9477-0fb85f0e70c0" containerName="payload-pre-processing" containerID="cri-o://4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331" gracePeriod=30
Sep 09 03:35:41.944100 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:41.944067    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"]
Sep 09 03:35:41.949438 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:41.949412    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7e9fa8-sgljf"]
Sep 09 03:35:42.052038 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.052013    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:42.055115 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.055093    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:42.190875 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.190842    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5hsn\" (UniqueName: \"kubernetes.io/projected/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-kube-api-access-v5hsn\") pod \"1044bb12-9ca3-4a04-90d7-3a5e23e3274c\" (UID: \"1044bb12-9ca3-4a04-90d7-3a5e23e3274c\") "
Sep 09 03:35:42.191084 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.190962    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbmdc\" (UniqueName: \"kubernetes.io/projected/53365a77-484a-4820-9477-0fb85f0e70c0-kube-api-access-bbmdc\") pod \"53365a77-484a-4820-9477-0fb85f0e70c0\" (UID: \"53365a77-484a-4820-9477-0fb85f0e70c0\") "
Sep 09 03:35:42.191084 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.191029    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53365a77-484a-4820-9477-0fb85f0e70c0-plugins-config-volume\") pod \"53365a77-484a-4820-9477-0fb85f0e70c0\" (UID: \"53365a77-484a-4820-9477-0fb85f0e70c0\") "
Sep 09 03:35:42.191084 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.191066    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-plugins-config-volume\") pod \"1044bb12-9ca3-4a04-90d7-3a5e23e3274c\" (UID: \"1044bb12-9ca3-4a04-90d7-3a5e23e3274c\") "
Sep 09 03:35:42.191511 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.191464    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53365a77-484a-4820-9477-0fb85f0e70c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "53365a77-484a-4820-9477-0fb85f0e70c0" (UID: "53365a77-484a-4820-9477-0fb85f0e70c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:35:42.191841 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.191597    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1044bb12-9ca3-4a04-90d7-3a5e23e3274c" (UID: "1044bb12-9ca3-4a04-90d7-3a5e23e3274c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:35:42.193189 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.193161    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53365a77-484a-4820-9477-0fb85f0e70c0-kube-api-access-bbmdc" (OuterVolumeSpecName: "kube-api-access-bbmdc") pod "53365a77-484a-4820-9477-0fb85f0e70c0" (UID: "53365a77-484a-4820-9477-0fb85f0e70c0"). InnerVolumeSpecName "kube-api-access-bbmdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:35:42.193361 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.193269    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-kube-api-access-v5hsn" (OuterVolumeSpecName: "kube-api-access-v5hsn") pod "1044bb12-9ca3-4a04-90d7-3a5e23e3274c" (UID: "1044bb12-9ca3-4a04-90d7-3a5e23e3274c"). InnerVolumeSpecName "kube-api-access-v5hsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:35:42.292537 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.292452    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbmdc\" (UniqueName: \"kubernetes.io/projected/53365a77-484a-4820-9477-0fb85f0e70c0-kube-api-access-bbmdc\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:35:42.292537 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.292484    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53365a77-484a-4820-9477-0fb85f0e70c0-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:35:42.292537 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.292494    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:35:42.292537 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.292502    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5hsn\" (UniqueName: \"kubernetes.io/projected/1044bb12-9ca3-4a04-90d7-3a5e23e3274c-kube-api-access-v5hsn\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:35:42.338216 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.338177    2580 generic.go:356] "Generic (PLEG): container finished" podID="53365a77-484a-4820-9477-0fb85f0e70c0" containerID="4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331" exitCode=0
Sep 09 03:35:42.338396 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.338222    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"
Sep 09 03:35:42.338396 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.338260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf" event={"ID":"53365a77-484a-4820-9477-0fb85f0e70c0","Type":"ContainerDied","Data":"4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331"}
Sep 09 03:35:42.338396 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.338319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf" event={"ID":"53365a77-484a-4820-9477-0fb85f0e70c0","Type":"ContainerDied","Data":"c028867f6fb54670bf5a129c84a9411bf3ac076162435c4a18948643ba2eac7b"}
Sep 09 03:35:42.338396 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.338349    2580 scope.go:117] "RemoveContainer" containerID="4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331"
Sep 09 03:35:42.339949 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.339925    2580 generic.go:356] "Generic (PLEG): container finished" podID="1044bb12-9ca3-4a04-90d7-3a5e23e3274c" containerID="772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226" exitCode=0
Sep 09 03:35:42.340077 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.339969    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"
Sep 09 03:35:42.340077 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.340019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k" event={"ID":"1044bb12-9ca3-4a04-90d7-3a5e23e3274c","Type":"ContainerDied","Data":"772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226"}
Sep 09 03:35:42.340077 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.340063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k" event={"ID":"1044bb12-9ca3-4a04-90d7-3a5e23e3274c","Type":"ContainerDied","Data":"62c3056c6fe9b7f39f25af039ba8c1680fc2c83428437cf13e8267cad78ef12d"}
Sep 09 03:35:42.348238 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.348216    2580 scope.go:117] "RemoveContainer" containerID="4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331"
Sep 09 03:35:42.348533 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:35:42.348514    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331\": container with ID starting with 4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331 not found: ID does not exist" containerID="4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331"
Sep 09 03:35:42.348649 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.348544    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331"} err="failed to get container status \"4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331\": rpc error: code = NotFound desc = could not find container \"4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331\": container with ID starting with 4f0ac462376e2134032e4c94cfd2b3d32609b59c82baf3e094532a5d57aa1331 not found: ID does not exist"
Sep 09 03:35:42.348649 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.348565    2580 scope.go:117] "RemoveContainer" containerID="772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226"
Sep 09 03:35:42.356771 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.356753    2580 scope.go:117] "RemoveContainer" containerID="772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226"
Sep 09 03:35:42.357085 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:35:42.357068    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226\": container with ID starting with 772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226 not found: ID does not exist" containerID="772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226"
Sep 09 03:35:42.357130 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.357091    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226"} err="failed to get container status \"772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226\": rpc error: code = NotFound desc = could not find container \"772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226\": container with ID starting with 772b6ad00d18e6158fc2973671b6f835e192efd1bbf159ed897121de7ba82226 not found: ID does not exist"
Sep 09 03:35:42.362240 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.362218    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"]
Sep 09 03:35:42.366486 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.366466    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7e9fa8-6c9dc899c8-7m9qf"]
Sep 09 03:35:42.382203 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.382177    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"]
Sep 09 03:35:42.384383 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.384361    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7e9fa8-5945b4f4c-twr6k"]
Sep 09 03:35:42.970399 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.970364    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1044bb12-9ca3-4a04-90d7-3a5e23e3274c" path="/var/lib/kubelet/pods/1044bb12-9ca3-4a04-90d7-3a5e23e3274c/volumes"
Sep 09 03:35:42.970891 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.970858    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53365a77-484a-4820-9477-0fb85f0e70c0" path="/var/lib/kubelet/pods/53365a77-484a-4820-9477-0fb85f0e70c0/volumes"
Sep 09 03:35:42.971208 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:42.971189    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c95f54-7c9d-4a97-9f43-bf844278e765" path="/var/lib/kubelet/pods/a1c95f54-7c9d-4a97-9f43-bf844278e765/volumes"
Sep 09 03:35:46.320663 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:46.320628    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:35:52.327698 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.327662    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"]
Sep 09 03:35:52.328165 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.328044    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53365a77-484a-4820-9477-0fb85f0e70c0" containerName="payload-pre-processing"
Sep 09 03:35:52.328165 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.328055    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53365a77-484a-4820-9477-0fb85f0e70c0" containerName="payload-pre-processing"
Sep 09 03:35:52.328165 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.328063    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c95f54-7c9d-4a97-9f43-bf844278e765" containerName="revoke-keys"
Sep 09 03:35:52.328165 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.328069    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c95f54-7c9d-4a97-9f43-bf844278e765" containerName="revoke-keys"
Sep 09 03:35:52.328165 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.328093    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1044bb12-9ca3-4a04-90d7-3a5e23e3274c" containerName="payload-processing"
Sep 09 03:35:52.328165 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.328098    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1044bb12-9ca3-4a04-90d7-3a5e23e3274c" containerName="payload-processing"
Sep 09 03:35:52.328165 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.328150    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1044bb12-9ca3-4a04-90d7-3a5e23e3274c" containerName="payload-processing"
Sep 09 03:35:52.328165 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.328157    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53365a77-484a-4820-9477-0fb85f0e70c0" containerName="payload-pre-processing"
Sep 09 03:35:52.328165 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.328164    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c95f54-7c9d-4a97-9f43-bf844278e765" containerName="revoke-keys"
Sep 09 03:35:52.332813 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.332796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"
Sep 09 03:35:52.334882 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.334863    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:35:52.335271 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.335137    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:35:52.338468 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.338446    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"]
Sep 09 03:35:52.483053 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.483016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43cf446-f408-460f-aac5-5af324a9189f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg\" (UID: \"b43cf446-f408-460f-aac5-5af324a9189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"
Sep 09 03:35:52.584466 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.584379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43cf446-f408-460f-aac5-5af324a9189f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg\" (UID: \"b43cf446-f408-460f-aac5-5af324a9189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"
Sep 09 03:35:52.585006 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.584985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43cf446-f408-460f-aac5-5af324a9189f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg\" (UID: \"b43cf446-f408-460f-aac5-5af324a9189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"
Sep 09 03:35:52.643886 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.643858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"
Sep 09 03:35:52.914318 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:52.914295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"]
Sep 09 03:35:52.916844 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:35:52.916811    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43cf446_f408_460f_aac5_5af324a9189f.slice/crio-40b6cb513c069658220900f12ffac4ffc09b2a85c0977dee6813125832d0baa0 WatchSource:0}: Error finding container 40b6cb513c069658220900f12ffac4ffc09b2a85c0977dee6813125832d0baa0: Status 404 returned error can't find the container with id 40b6cb513c069658220900f12ffac4ffc09b2a85c0977dee6813125832d0baa0
Sep 09 03:35:53.401964 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:53.401926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg" event={"ID":"b43cf446-f408-460f-aac5-5af324a9189f","Type":"ContainerStarted","Data":"3762440d70fb8c8bae7591bd53645591b5833ab6b84f45c35737208f993d7952"}
Sep 09 03:35:53.402397 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:53.401978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg" event={"ID":"b43cf446-f408-460f-aac5-5af324a9189f","Type":"ContainerStarted","Data":"40b6cb513c069658220900f12ffac4ffc09b2a85c0977dee6813125832d0baa0"}
Sep 09 03:35:53.418024 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:53.417968    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg" podStartSLOduration=1.417954219 podStartE2EDuration="1.417954219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:35:53.415643414 +0000 UTC m=+1796.941958180" watchObservedRunningTime="2026-09-09 03:35:53.417954219 +0000 UTC m=+1796.944268955"
Sep 09 03:35:54.409881 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:54.409848    2580 generic.go:356] "Generic (PLEG): container finished" podID="b43cf446-f408-460f-aac5-5af324a9189f" containerID="3762440d70fb8c8bae7591bd53645591b5833ab6b84f45c35737208f993d7952" exitCode=0
Sep 09 03:35:54.410264 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:54.409895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg" event={"ID":"b43cf446-f408-460f-aac5-5af324a9189f","Type":"ContainerDied","Data":"3762440d70fb8c8bae7591bd53645591b5833ab6b84f45c35737208f993d7952"}
Sep 09 03:35:55.547736 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:55.547709    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"
Sep 09 03:35:55.610023 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:55.609988    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43cf446-f408-460f-aac5-5af324a9189f-maas-api-service-ca\") pod \"b43cf446-f408-460f-aac5-5af324a9189f\" (UID: \"b43cf446-f408-460f-aac5-5af324a9189f\") "
Sep 09 03:35:55.617205 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:55.610687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43cf446-f408-460f-aac5-5af324a9189f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b43cf446-f408-460f-aac5-5af324a9189f" (UID: "b43cf446-f408-460f-aac5-5af324a9189f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:35:55.711733 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:55.711654    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43cf446-f408-460f-aac5-5af324a9189f-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:35:56.421538 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:56.421504    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"
Sep 09 03:35:56.421735 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:56.421540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg" event={"ID":"b43cf446-f408-460f-aac5-5af324a9189f","Type":"ContainerDied","Data":"40b6cb513c069658220900f12ffac4ffc09b2a85c0977dee6813125832d0baa0"}
Sep 09 03:35:56.421735 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:35:56.421617    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40b6cb513c069658220900f12ffac4ffc09b2a85c0977dee6813125832d0baa0"
Sep 09 03:36:05.607615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:05.607562    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:36:07.988252 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:07.988218    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"]
Sep 09 03:36:07.988715 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:07.988428    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm" podUID="f12cbae4-cbb2-4292-8617-b908fa3d308e" containerName="payload-processing" containerID="cri-o://26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e" gracePeriod=30
Sep 09 03:36:08.000742 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.000711    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"]
Sep 09 03:36:08.001035 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.000988    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c" podUID="2f41d8ba-03b7-4409-bd3b-ec1b782e28fc" containerName="payload-pre-processing" containerID="cri-o://5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89" gracePeriod=30
Sep 09 03:36:08.283255 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.283230    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:36:08.286720 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.286702    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:36:08.327080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.327052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12cbae4-cbb2-4292-8617-b908fa3d308e-plugins-config-volume\") pod \"f12cbae4-cbb2-4292-8617-b908fa3d308e\" (UID: \"f12cbae4-cbb2-4292-8617-b908fa3d308e\") "
Sep 09 03:36:08.327221 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.327141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-plugins-config-volume\") pod \"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc\" (UID: \"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc\") "
Sep 09 03:36:08.327885 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.327435    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7v7x\" (UniqueName: \"kubernetes.io/projected/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-kube-api-access-k7v7x\") pod \"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc\" (UID: \"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc\") "
Sep 09 03:36:08.327885 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.327484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sktrw\" (UniqueName: \"kubernetes.io/projected/f12cbae4-cbb2-4292-8617-b908fa3d308e-kube-api-access-sktrw\") pod \"f12cbae4-cbb2-4292-8617-b908fa3d308e\" (UID: \"f12cbae4-cbb2-4292-8617-b908fa3d308e\") "
Sep 09 03:36:08.327885 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.327504    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12cbae4-cbb2-4292-8617-b908fa3d308e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f12cbae4-cbb2-4292-8617-b908fa3d308e" (UID: "f12cbae4-cbb2-4292-8617-b908fa3d308e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:36:08.327885 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.327518    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f41d8ba-03b7-4409-bd3b-ec1b782e28fc" (UID: "2f41d8ba-03b7-4409-bd3b-ec1b782e28fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:36:08.327885 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.327842    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12cbae4-cbb2-4292-8617-b908fa3d308e-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:36:08.327885 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.327862    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:36:08.330065 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.330028    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-kube-api-access-k7v7x" (OuterVolumeSpecName: "kube-api-access-k7v7x") pod "2f41d8ba-03b7-4409-bd3b-ec1b782e28fc" (UID: "2f41d8ba-03b7-4409-bd3b-ec1b782e28fc"). InnerVolumeSpecName "kube-api-access-k7v7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:36:08.330227 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.330081    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12cbae4-cbb2-4292-8617-b908fa3d308e-kube-api-access-sktrw" (OuterVolumeSpecName: "kube-api-access-sktrw") pod "f12cbae4-cbb2-4292-8617-b908fa3d308e" (UID: "f12cbae4-cbb2-4292-8617-b908fa3d308e"). InnerVolumeSpecName "kube-api-access-sktrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:36:08.428936 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.428901    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7v7x\" (UniqueName: \"kubernetes.io/projected/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc-kube-api-access-k7v7x\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:36:08.428936 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.428928    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sktrw\" (UniqueName: \"kubernetes.io/projected/f12cbae4-cbb2-4292-8617-b908fa3d308e-kube-api-access-sktrw\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:36:08.478298 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.478264    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"]
Sep 09 03:36:08.484270 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.484240    2580 generic.go:356] "Generic (PLEG): container finished" podID="f12cbae4-cbb2-4292-8617-b908fa3d308e" containerID="26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e" exitCode=0
Sep 09 03:36:08.484425 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.484283    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"
Sep 09 03:36:08.484425 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.484338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm" event={"ID":"f12cbae4-cbb2-4292-8617-b908fa3d308e","Type":"ContainerDied","Data":"26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e"}
Sep 09 03:36:08.484425 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.484407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm" event={"ID":"f12cbae4-cbb2-4292-8617-b908fa3d308e","Type":"ContainerDied","Data":"5f8be772b5dc161444eeba77d0d13d638d341ffe44448ef2d64388d3d353b16d"}
Sep 09 03:36:08.484547 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.484440    2580 scope.go:117] "RemoveContainer" containerID="26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e"
Sep 09 03:36:08.484547 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.484524    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed81bb-gn8kg"]
Sep 09 03:36:08.486363 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.486337    2580 generic.go:356] "Generic (PLEG): container finished" podID="2f41d8ba-03b7-4409-bd3b-ec1b782e28fc" containerID="5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89" exitCode=0
Sep 09 03:36:08.486486 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.486449    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"
Sep 09 03:36:08.486486 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.486472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c" event={"ID":"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc","Type":"ContainerDied","Data":"5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89"}
Sep 09 03:36:08.486568 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.486518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c" event={"ID":"2f41d8ba-03b7-4409-bd3b-ec1b782e28fc","Type":"ContainerDied","Data":"7ece42746a877b0dac9711327ad014db4f77698a842801ace6e6af67f58632c3"}
Sep 09 03:36:08.496469 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.496423    2580 scope.go:117] "RemoveContainer" containerID="26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e"
Sep 09 03:36:08.497077 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:36:08.497040    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e\": container with ID starting with 26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e not found: ID does not exist" containerID="26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e"
Sep 09 03:36:08.497179 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.497078    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e"} err="failed to get container status \"26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e\": rpc error: code = NotFound desc = could not find container \"26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e\": container with ID starting with 26c043af710a63c871a68c195334aabbad5a2c4460646da5a46e333326e59c2e not found: ID does not exist"
Sep 09 03:36:08.497179 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.497095    2580 scope.go:117] "RemoveContainer" containerID="5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89"
Sep 09 03:36:08.505725 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.505708    2580 scope.go:117] "RemoveContainer" containerID="5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89"
Sep 09 03:36:08.505978 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:36:08.505962    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89\": container with ID starting with 5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89 not found: ID does not exist" containerID="5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89"
Sep 09 03:36:08.506018 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.505986    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89"} err="failed to get container status \"5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89\": rpc error: code = NotFound desc = could not find container \"5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89\": container with ID starting with 5cbd63861c583bad928a1c85f2a438eb14a26829235a09163fe8bd115a415f89 not found: ID does not exist"
Sep 09 03:36:08.518100 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.518037    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"]
Sep 09 03:36:08.524705 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.524679    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed81bb-7cd9997fcd-4cd4c"]
Sep 09 03:36:08.537782 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.537760    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"]
Sep 09 03:36:08.548393 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.548357    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed81bb-69f8fd7f85-9zgrm"]
Sep 09 03:36:08.970687 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.970652    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f41d8ba-03b7-4409-bd3b-ec1b782e28fc" path="/var/lib/kubelet/pods/2f41d8ba-03b7-4409-bd3b-ec1b782e28fc/volumes"
Sep 09 03:36:08.971010 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.970997    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b43cf446-f408-460f-aac5-5af324a9189f" path="/var/lib/kubelet/pods/b43cf446-f408-460f-aac5-5af324a9189f/volumes"
Sep 09 03:36:08.971303 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:08.971292    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12cbae4-cbb2-4292-8617-b908fa3d308e" path="/var/lib/kubelet/pods/f12cbae4-cbb2-4292-8617-b908fa3d308e/volumes"
Sep 09 03:36:19.300399 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:19.300352    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:36:57.427660 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.427629    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"]
Sep 09 03:36:57.428188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.428160    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12cbae4-cbb2-4292-8617-b908fa3d308e" containerName="payload-processing"
Sep 09 03:36:57.428188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.428185    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12cbae4-cbb2-4292-8617-b908fa3d308e" containerName="payload-processing"
Sep 09 03:36:57.428327 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.428215    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43cf446-f408-460f-aac5-5af324a9189f" containerName="revoke-keys"
Sep 09 03:36:57.428327 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.428225    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43cf446-f408-460f-aac5-5af324a9189f" containerName="revoke-keys"
Sep 09 03:36:57.428327 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.428246    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f41d8ba-03b7-4409-bd3b-ec1b782e28fc" containerName="payload-pre-processing"
Sep 09 03:36:57.428327 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.428255    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f41d8ba-03b7-4409-bd3b-ec1b782e28fc" containerName="payload-pre-processing"
Sep 09 03:36:57.428524 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.428366    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43cf446-f408-460f-aac5-5af324a9189f" containerName="revoke-keys"
Sep 09 03:36:57.428524 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.428382    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12cbae4-cbb2-4292-8617-b908fa3d308e" containerName="payload-processing"
Sep 09 03:36:57.428524 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.428395    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f41d8ba-03b7-4409-bd3b-ec1b782e28fc" containerName="payload-pre-processing"
Sep 09 03:36:57.432993 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.432975    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"
Sep 09 03:36:57.435246 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.435219    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:36:57.435375 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.435327    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:36:57.439077 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.439054    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"]
Sep 09 03:36:57.578019 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.577983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8605de5c-fb63-44ce-9e7b-46fc65e1c563-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp\" (UID: \"8605de5c-fb63-44ce-9e7b-46fc65e1c563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"
Sep 09 03:36:57.679549 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.679515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8605de5c-fb63-44ce-9e7b-46fc65e1c563-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp\" (UID: \"8605de5c-fb63-44ce-9e7b-46fc65e1c563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"
Sep 09 03:36:57.680171 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.680148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8605de5c-fb63-44ce-9e7b-46fc65e1c563-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp\" (UID: \"8605de5c-fb63-44ce-9e7b-46fc65e1c563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"
Sep 09 03:36:57.745487 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:57.745404    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"
Sep 09 03:36:58.018033 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:58.018006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"]
Sep 09 03:36:58.020115 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:36:58.020085    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8605de5c_fb63_44ce_9e7b_46fc65e1c563.slice/crio-53228c2e1e920b7fd0440969e552d98d50ca56ec6c0df22e4f68f87a14193b23 WatchSource:0}: Error finding container 53228c2e1e920b7fd0440969e552d98d50ca56ec6c0df22e4f68f87a14193b23: Status 404 returned error can't find the container with id 53228c2e1e920b7fd0440969e552d98d50ca56ec6c0df22e4f68f87a14193b23
Sep 09 03:36:58.739300 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:58.739223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp" event={"ID":"8605de5c-fb63-44ce-9e7b-46fc65e1c563","Type":"ContainerStarted","Data":"83e42f0b268871f8a4003ada631bc5758f601255bd475400bcf1e40dbc709685"}
Sep 09 03:36:58.739300 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:58.739277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp" event={"ID":"8605de5c-fb63-44ce-9e7b-46fc65e1c563","Type":"ContainerStarted","Data":"53228c2e1e920b7fd0440969e552d98d50ca56ec6c0df22e4f68f87a14193b23"}
Sep 09 03:36:58.756725 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:58.756676    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp" podStartSLOduration=1.756658718 podStartE2EDuration="1.756658718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:36:58.75436165 +0000 UTC m=+1862.280676393" watchObservedRunningTime="2026-09-09 03:36:58.756658718 +0000 UTC m=+1862.282973497"
Sep 09 03:36:59.746443 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:59.746406    2580 generic.go:356] "Generic (PLEG): container finished" podID="8605de5c-fb63-44ce-9e7b-46fc65e1c563" containerID="83e42f0b268871f8a4003ada631bc5758f601255bd475400bcf1e40dbc709685" exitCode=0
Sep 09 03:36:59.746443 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:36:59.746448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp" event={"ID":"8605de5c-fb63-44ce-9e7b-46fc65e1c563","Type":"ContainerDied","Data":"83e42f0b268871f8a4003ada631bc5758f601255bd475400bcf1e40dbc709685"}
Sep 09 03:37:00.883527 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:00.883504    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"
Sep 09 03:37:01.014261 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:01.014169    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8605de5c-fb63-44ce-9e7b-46fc65e1c563-maas-api-service-ca\") pod \"8605de5c-fb63-44ce-9e7b-46fc65e1c563\" (UID: \"8605de5c-fb63-44ce-9e7b-46fc65e1c563\") "
Sep 09 03:37:01.014609 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:01.014563    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8605de5c-fb63-44ce-9e7b-46fc65e1c563-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8605de5c-fb63-44ce-9e7b-46fc65e1c563" (UID: "8605de5c-fb63-44ce-9e7b-46fc65e1c563"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:01.115873 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:01.115836    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8605de5c-fb63-44ce-9e7b-46fc65e1c563-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:37:01.757591 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:01.757551    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"
Sep 09 03:37:01.757591 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:01.757557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp" event={"ID":"8605de5c-fb63-44ce-9e7b-46fc65e1c563","Type":"ContainerDied","Data":"53228c2e1e920b7fd0440969e552d98d50ca56ec6c0df22e4f68f87a14193b23"}
Sep 09 03:37:01.757803 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:01.757618    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53228c2e1e920b7fd0440969e552d98d50ca56ec6c0df22e4f68f87a14193b23"
Sep 09 03:37:08.395488 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:08.395447    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"]
Sep 09 03:37:08.399885 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:08.399852    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aacd65-2rzwp"]
Sep 09 03:37:08.971033 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:08.970997    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8605de5c-fb63-44ce-9e7b-46fc65e1c563" path="/var/lib/kubelet/pods/8605de5c-fb63-44ce-9e7b-46fc65e1c563/volumes"
Sep 09 03:37:11.434704 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:11.434667    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:37:15.262500 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.262463    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"]
Sep 09 03:37:15.262880 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.262858    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8605de5c-fb63-44ce-9e7b-46fc65e1c563" containerName="revoke-keys"
Sep 09 03:37:15.262880 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.262869    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8605de5c-fb63-44ce-9e7b-46fc65e1c563" containerName="revoke-keys"
Sep 09 03:37:15.262967 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.262938    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8605de5c-fb63-44ce-9e7b-46fc65e1c563" containerName="revoke-keys"
Sep 09 03:37:15.265809 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.265793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"
Sep 09 03:37:15.268268 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.268241    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:37:15.268381 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.268293    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:37:15.273928 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.273899    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"]
Sep 09 03:37:15.352944 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.352913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c74c05-6558-4c12-a55b-b9f6c7213c70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk\" (UID: \"f6c74c05-6558-4c12-a55b-b9f6c7213c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"
Sep 09 03:37:15.454001 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.453960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c74c05-6558-4c12-a55b-b9f6c7213c70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk\" (UID: \"f6c74c05-6558-4c12-a55b-b9f6c7213c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"
Sep 09 03:37:15.454639 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.454618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c74c05-6558-4c12-a55b-b9f6c7213c70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk\" (UID: \"f6c74c05-6558-4c12-a55b-b9f6c7213c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"
Sep 09 03:37:15.577400 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.577314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"
Sep 09 03:37:15.852895 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:15.852868    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"]
Sep 09 03:37:15.855892 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:37:15.855858    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c74c05_6558_4c12_a55b_b9f6c7213c70.slice/crio-c3e5d964dd1e3c23b7c575964b5558ed1de8079ebac28610e2d0187b58971619 WatchSource:0}: Error finding container c3e5d964dd1e3c23b7c575964b5558ed1de8079ebac28610e2d0187b58971619: Status 404 returned error can't find the container with id c3e5d964dd1e3c23b7c575964b5558ed1de8079ebac28610e2d0187b58971619
Sep 09 03:37:16.836134 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:16.836045    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6c74c05-6558-4c12-a55b-b9f6c7213c70" containerID="fcf336b0c3de7ad53e9b4ddf4f362e8433a95ff8830405175a5f064ce38e4ee7" exitCode=0
Sep 09 03:37:16.836134 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:16.836091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk" event={"ID":"f6c74c05-6558-4c12-a55b-b9f6c7213c70","Type":"ContainerDied","Data":"fcf336b0c3de7ad53e9b4ddf4f362e8433a95ff8830405175a5f064ce38e4ee7"}
Sep 09 03:37:16.836540 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:16.836138    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk" event={"ID":"f6c74c05-6558-4c12-a55b-b9f6c7213c70","Type":"ContainerStarted","Data":"c3e5d964dd1e3c23b7c575964b5558ed1de8079ebac28610e2d0187b58971619"}
Sep 09 03:37:17.967304 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:17.967279    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"
Sep 09 03:37:18.076325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:18.076284    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c74c05-6558-4c12-a55b-b9f6c7213c70-maas-api-service-ca\") pod \"f6c74c05-6558-4c12-a55b-b9f6c7213c70\" (UID: \"f6c74c05-6558-4c12-a55b-b9f6c7213c70\") "
Sep 09 03:37:18.076702 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:18.076674    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c74c05-6558-4c12-a55b-b9f6c7213c70-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6c74c05-6558-4c12-a55b-b9f6c7213c70" (UID: "f6c74c05-6558-4c12-a55b-b9f6c7213c70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:18.177744 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:18.177711    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c74c05-6558-4c12-a55b-b9f6c7213c70-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:37:18.847714 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:18.847679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk" event={"ID":"f6c74c05-6558-4c12-a55b-b9f6c7213c70","Type":"ContainerDied","Data":"c3e5d964dd1e3c23b7c575964b5558ed1de8079ebac28610e2d0187b58971619"}
Sep 09 03:37:18.847714 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:18.847703    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"
Sep 09 03:37:18.847935 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:18.847757    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3e5d964dd1e3c23b7c575964b5558ed1de8079ebac28610e2d0187b58971619"
Sep 09 03:37:25.750973 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:25.750940    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm"]
Sep 09 03:37:25.753657 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:25.753631    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1a85fa-29815410-h2ktm"]
Sep 09 03:37:25.774197 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:25.774171    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"]
Sep 09 03:37:25.774469 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:25.774414    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw" podUID="561c0b32-a898-4b89-b602-453f493a1812" containerName="payload-processing" containerID="cri-o://1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a" gracePeriod=30
Sep 09 03:37:25.784041 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:25.784019    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"]
Sep 09 03:37:25.784321 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:25.784263    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m" podUID="571d961e-00a1-4701-80ee-57161ac83c04" containerName="payload-pre-processing" containerID="cri-o://405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2" gracePeriod=30
Sep 09 03:37:25.994541 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:25.994503    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"]
Sep 09 03:37:25.996164 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:25.996132    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1a85fa-7z9hk"]
Sep 09 03:37:26.060560 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.060537    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:37:26.063478 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.063459    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:37:26.148546 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.148512    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72ffd\" (UniqueName: \"kubernetes.io/projected/561c0b32-a898-4b89-b602-453f493a1812-kube-api-access-72ffd\") pod \"561c0b32-a898-4b89-b602-453f493a1812\" (UID: \"561c0b32-a898-4b89-b602-453f493a1812\") "
Sep 09 03:37:26.148546 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.148548    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571d961e-00a1-4701-80ee-57161ac83c04-plugins-config-volume\") pod \"571d961e-00a1-4701-80ee-57161ac83c04\" (UID: \"571d961e-00a1-4701-80ee-57161ac83c04\") "
Sep 09 03:37:26.148789 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.148570    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561c0b32-a898-4b89-b602-453f493a1812-plugins-config-volume\") pod \"561c0b32-a898-4b89-b602-453f493a1812\" (UID: \"561c0b32-a898-4b89-b602-453f493a1812\") "
Sep 09 03:37:26.148789 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.148639    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwkkh\" (UniqueName: \"kubernetes.io/projected/571d961e-00a1-4701-80ee-57161ac83c04-kube-api-access-vwkkh\") pod \"571d961e-00a1-4701-80ee-57161ac83c04\" (UID: \"571d961e-00a1-4701-80ee-57161ac83c04\") "
Sep 09 03:37:26.148996 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.148961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561c0b32-a898-4b89-b602-453f493a1812-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "561c0b32-a898-4b89-b602-453f493a1812" (UID: "561c0b32-a898-4b89-b602-453f493a1812"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:26.148996 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.148976    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571d961e-00a1-4701-80ee-57161ac83c04-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "571d961e-00a1-4701-80ee-57161ac83c04" (UID: "571d961e-00a1-4701-80ee-57161ac83c04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:26.150882 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.150848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561c0b32-a898-4b89-b602-453f493a1812-kube-api-access-72ffd" (OuterVolumeSpecName: "kube-api-access-72ffd") pod "561c0b32-a898-4b89-b602-453f493a1812" (UID: "561c0b32-a898-4b89-b602-453f493a1812"). InnerVolumeSpecName "kube-api-access-72ffd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:26.150982 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.150922    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571d961e-00a1-4701-80ee-57161ac83c04-kube-api-access-vwkkh" (OuterVolumeSpecName: "kube-api-access-vwkkh") pod "571d961e-00a1-4701-80ee-57161ac83c04" (UID: "571d961e-00a1-4701-80ee-57161ac83c04"). InnerVolumeSpecName "kube-api-access-vwkkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:26.249404 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.249361    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72ffd\" (UniqueName: \"kubernetes.io/projected/561c0b32-a898-4b89-b602-453f493a1812-kube-api-access-72ffd\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:37:26.249404 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.249396    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571d961e-00a1-4701-80ee-57161ac83c04-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:37:26.249404 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.249406    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561c0b32-a898-4b89-b602-453f493a1812-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:37:26.249683 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.249416    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwkkh\" (UniqueName: \"kubernetes.io/projected/571d961e-00a1-4701-80ee-57161ac83c04-kube-api-access-vwkkh\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:37:26.889062 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.889029    2580 generic.go:356] "Generic (PLEG): container finished" podID="571d961e-00a1-4701-80ee-57161ac83c04" containerID="405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2" exitCode=0
Sep 09 03:37:26.889504 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.889076    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"
Sep 09 03:37:26.889504 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.889121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m" event={"ID":"571d961e-00a1-4701-80ee-57161ac83c04","Type":"ContainerDied","Data":"405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2"}
Sep 09 03:37:26.889504 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.889180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m" event={"ID":"571d961e-00a1-4701-80ee-57161ac83c04","Type":"ContainerDied","Data":"237c65a27be8b76b50b73e412da640a24785b6a057174161bc5b9ab38500ca7d"}
Sep 09 03:37:26.889504 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.889217    2580 scope.go:117] "RemoveContainer" containerID="405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2"
Sep 09 03:37:26.890754 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.890733    2580 generic.go:356] "Generic (PLEG): container finished" podID="561c0b32-a898-4b89-b602-453f493a1812" containerID="1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a" exitCode=0
Sep 09 03:37:26.890846 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.890764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw" event={"ID":"561c0b32-a898-4b89-b602-453f493a1812","Type":"ContainerDied","Data":"1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a"}
Sep 09 03:37:26.890846 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.890778    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"
Sep 09 03:37:26.890846 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.890826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw" event={"ID":"561c0b32-a898-4b89-b602-453f493a1812","Type":"ContainerDied","Data":"8282857dcb85bc83b8535e647c301c636dbdee109c784e0c7f840b1958f532c0"}
Sep 09 03:37:26.898883 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.898861    2580 scope.go:117] "RemoveContainer" containerID="405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2"
Sep 09 03:37:26.899166 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:37:26.899148    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2\": container with ID starting with 405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2 not found: ID does not exist" containerID="405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2"
Sep 09 03:37:26.899253 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.899171    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2"} err="failed to get container status \"405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2\": rpc error: code = NotFound desc = could not find container \"405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2\": container with ID starting with 405f6330a5a33675b40b5e67df73de00178f20cfac75b37f821ed190db0753f2 not found: ID does not exist"
Sep 09 03:37:26.899253 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.899188    2580 scope.go:117] "RemoveContainer" containerID="1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a"
Sep 09 03:37:26.907206 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.907184    2580 scope.go:117] "RemoveContainer" containerID="1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a"
Sep 09 03:37:26.907472 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:37:26.907453    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a\": container with ID starting with 1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a not found: ID does not exist" containerID="1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a"
Sep 09 03:37:26.907557 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.907479    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a"} err="failed to get container status \"1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a\": rpc error: code = NotFound desc = could not find container \"1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a\": container with ID starting with 1841e57de618517b02f2f30d27c2077a731c66898b6662c7c37e306f3df8f03a not found: ID does not exist"
Sep 09 03:37:26.913124 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.913103    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"]
Sep 09 03:37:26.917709 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.917689    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1a85fa-686848c7f4-6tx7m"]
Sep 09 03:37:26.928418 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.928393    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"]
Sep 09 03:37:26.934434 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.934409    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1a85fa-5f9c4db69f-h78fw"]
Sep 09 03:37:26.970984 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.970956    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="561c0b32-a898-4b89-b602-453f493a1812" path="/var/lib/kubelet/pods/561c0b32-a898-4b89-b602-453f493a1812/volumes"
Sep 09 03:37:26.971325 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.971312    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="571d961e-00a1-4701-80ee-57161ac83c04" path="/var/lib/kubelet/pods/571d961e-00a1-4701-80ee-57161ac83c04/volumes"
Sep 09 03:37:26.971549 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.971537    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b33b87-e546-4134-a382-09c78364f772" path="/var/lib/kubelet/pods/a2b33b87-e546-4134-a382-09c78364f772/volumes"
Sep 09 03:37:26.971794 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:26.971782    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6c74c05-6558-4c12-a55b-b9f6c7213c70" path="/var/lib/kubelet/pods/f6c74c05-6558-4c12-a55b-b9f6c7213c70/volumes"
Sep 09 03:37:28.961464 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:28.961427    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:37:32.987740 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.987707    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"]
Sep 09 03:37:32.988188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.988028    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="561c0b32-a898-4b89-b602-453f493a1812" containerName="payload-processing"
Sep 09 03:37:32.988188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.988039    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="561c0b32-a898-4b89-b602-453f493a1812" containerName="payload-processing"
Sep 09 03:37:32.988188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.988052    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="571d961e-00a1-4701-80ee-57161ac83c04" containerName="payload-pre-processing"
Sep 09 03:37:32.988188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.988057    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="571d961e-00a1-4701-80ee-57161ac83c04" containerName="payload-pre-processing"
Sep 09 03:37:32.988188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.988069    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c74c05-6558-4c12-a55b-b9f6c7213c70" containerName="revoke-keys"
Sep 09 03:37:32.988188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.988075    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c74c05-6558-4c12-a55b-b9f6c7213c70" containerName="revoke-keys"
Sep 09 03:37:32.988188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.988131    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="561c0b32-a898-4b89-b602-453f493a1812" containerName="payload-processing"
Sep 09 03:37:32.988188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.988138    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c74c05-6558-4c12-a55b-b9f6c7213c70" containerName="revoke-keys"
Sep 09 03:37:32.988188 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.988144    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="571d961e-00a1-4701-80ee-57161ac83c04" containerName="payload-pre-processing"
Sep 09 03:37:32.991160 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.991144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"
Sep 09 03:37:32.993394 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.993376    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:37:32.993495 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.993399    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:37:32.998608 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:32.998565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"]
Sep 09 03:37:33.111539 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:33.111492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0325d57b-a494-4db4-ab84-32e2ad4308b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb\" (UID: \"0325d57b-a494-4db4-ab84-32e2ad4308b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"
Sep 09 03:37:33.212793 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:33.212754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0325d57b-a494-4db4-ab84-32e2ad4308b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb\" (UID: \"0325d57b-a494-4db4-ab84-32e2ad4308b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"
Sep 09 03:37:33.213403 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:33.213380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0325d57b-a494-4db4-ab84-32e2ad4308b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb\" (UID: \"0325d57b-a494-4db4-ab84-32e2ad4308b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"
Sep 09 03:37:33.303420 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:33.303320    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"
Sep 09 03:37:33.581885 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:37:33.581851    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0325d57b_a494_4db4_ab84_32e2ad4308b2.slice/crio-310b52e6aa78e89af2907ac807c1a310607b37a342e4ac6b7bbd23960eec3121 WatchSource:0}: Error finding container 310b52e6aa78e89af2907ac807c1a310607b37a342e4ac6b7bbd23960eec3121: Status 404 returned error can't find the container with id 310b52e6aa78e89af2907ac807c1a310607b37a342e4ac6b7bbd23960eec3121
Sep 09 03:37:33.582656 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:33.582636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"]
Sep 09 03:37:33.930162 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:33.930122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb" event={"ID":"0325d57b-a494-4db4-ab84-32e2ad4308b2","Type":"ContainerStarted","Data":"c595e25429f9665799674a25ee585e7feba9f8a97c54c579700b6452483befe9"}
Sep 09 03:37:33.930350 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:33.930177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb" event={"ID":"0325d57b-a494-4db4-ab84-32e2ad4308b2","Type":"ContainerStarted","Data":"310b52e6aa78e89af2907ac807c1a310607b37a342e4ac6b7bbd23960eec3121"}
Sep 09 03:37:33.946986 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:33.946941    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb" podStartSLOduration=1.946926004 podStartE2EDuration="1.946926004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:37:33.945531036 +0000 UTC m=+1897.471845770" watchObservedRunningTime="2026-09-09 03:37:33.946926004 +0000 UTC m=+1897.473240740"
Sep 09 03:37:34.937138 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:34.937099    2580 generic.go:356] "Generic (PLEG): container finished" podID="0325d57b-a494-4db4-ab84-32e2ad4308b2" containerID="c595e25429f9665799674a25ee585e7feba9f8a97c54c579700b6452483befe9" exitCode=0
Sep 09 03:37:34.937548 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:34.937170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb" event={"ID":"0325d57b-a494-4db4-ab84-32e2ad4308b2","Type":"ContainerDied","Data":"c595e25429f9665799674a25ee585e7feba9f8a97c54c579700b6452483befe9"}
Sep 09 03:37:36.075989 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:36.075961    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"
Sep 09 03:37:36.139950 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:36.139920    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0325d57b-a494-4db4-ab84-32e2ad4308b2-maas-api-service-ca\") pod \"0325d57b-a494-4db4-ab84-32e2ad4308b2\" (UID: \"0325d57b-a494-4db4-ab84-32e2ad4308b2\") "
Sep 09 03:37:36.140300 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:36.140277    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0325d57b-a494-4db4-ab84-32e2ad4308b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0325d57b-a494-4db4-ab84-32e2ad4308b2" (UID: "0325d57b-a494-4db4-ab84-32e2ad4308b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:36.241455 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:36.241373    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0325d57b-a494-4db4-ab84-32e2ad4308b2-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:37:36.949924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:36.949894    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"
Sep 09 03:37:36.950121 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:36.949893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb" event={"ID":"0325d57b-a494-4db4-ab84-32e2ad4308b2","Type":"ContainerDied","Data":"310b52e6aa78e89af2907ac807c1a310607b37a342e4ac6b7bbd23960eec3121"}
Sep 09 03:37:36.950121 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:36.950045    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="310b52e6aa78e89af2907ac807c1a310607b37a342e4ac6b7bbd23960eec3121"
Sep 09 03:37:43.595569 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.595530    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc"]
Sep 09 03:37:43.599731 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.599702    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cd1da6-29815410-2zgsc"]
Sep 09 03:37:43.622511 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.622479    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"]
Sep 09 03:37:43.622791 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.622757    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h" podUID="a864a82a-6cbc-453e-9a90-9d7366671883" containerName="payload-processing" containerID="cri-o://6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d" gracePeriod=30
Sep 09 03:37:43.631722 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.630396    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"]
Sep 09 03:37:43.631722 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.630651    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699" podUID="cc83b8e1-ad9e-4cfd-8c3d-76a048762908" containerName="payload-pre-processing" containerID="cri-o://a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d" gracePeriod=30
Sep 09 03:37:43.798509 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.798477    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"]
Sep 09 03:37:43.804508 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.804478    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cd1da6-vcnmb"]
Sep 09 03:37:43.901521 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.901500    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:37:43.904428 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.904411    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:37:43.986788 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.986751    2580 generic.go:356] "Generic (PLEG): container finished" podID="a864a82a-6cbc-453e-9a90-9d7366671883" containerID="6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d" exitCode=0
Sep 09 03:37:43.986978 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.986797    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"
Sep 09 03:37:43.986978 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.986844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h" event={"ID":"a864a82a-6cbc-453e-9a90-9d7366671883","Type":"ContainerDied","Data":"6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d"}
Sep 09 03:37:43.986978 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.986893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h" event={"ID":"a864a82a-6cbc-453e-9a90-9d7366671883","Type":"ContainerDied","Data":"1c207ec9c1c7ee11c3877b717923a3a4813425701e1ada3ef1c764fd0dbd6904"}
Sep 09 03:37:43.986978 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.986936    2580 scope.go:117] "RemoveContainer" containerID="6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d"
Sep 09 03:37:43.988302 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.988275    2580 generic.go:356] "Generic (PLEG): container finished" podID="cc83b8e1-ad9e-4cfd-8c3d-76a048762908" containerID="a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d" exitCode=0
Sep 09 03:37:43.988441 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.988311    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"
Sep 09 03:37:43.988441 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.988314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699" event={"ID":"cc83b8e1-ad9e-4cfd-8c3d-76a048762908","Type":"ContainerDied","Data":"a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d"}
Sep 09 03:37:43.988524 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.988467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699" event={"ID":"cc83b8e1-ad9e-4cfd-8c3d-76a048762908","Type":"ContainerDied","Data":"a9475cb9554718b009c461c7af51ffeeffb06f1d6073a08bc4c82351df96cb8a"}
Sep 09 03:37:43.996367 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.996346    2580 scope.go:117] "RemoveContainer" containerID="6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d"
Sep 09 03:37:43.996629 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:37:43.996605    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d\": container with ID starting with 6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d not found: ID does not exist" containerID="6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d"
Sep 09 03:37:43.996741 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.996636    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d"} err="failed to get container status \"6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d\": rpc error: code = NotFound desc = could not find container \"6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d\": container with ID starting with 6f70575f4df549c3382a6c38edebdfa4d7f4a39e5924eb94e83d2337e86c537d not found: ID does not exist"
Sep 09 03:37:43.996741 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:43.996653    2580 scope.go:117] "RemoveContainer" containerID="a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d"
Sep 09 03:37:44.004506 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.004490    2580 scope.go:117] "RemoveContainer" containerID="a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d"
Sep 09 03:37:44.004807 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:37:44.004789    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d\": container with ID starting with a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d not found: ID does not exist" containerID="a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d"
Sep 09 03:37:44.004874 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.004812    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d"} err="failed to get container status \"a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d\": rpc error: code = NotFound desc = could not find container \"a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d\": container with ID starting with a8bf3f0830ae1e16a23e21b057b621ce5392211283f15fec978b9fb0602ace3d not found: ID does not exist"
Sep 09 03:37:44.014080 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.014060    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a864a82a-6cbc-453e-9a90-9d7366671883-plugins-config-volume\") pod \"a864a82a-6cbc-453e-9a90-9d7366671883\" (UID: \"a864a82a-6cbc-453e-9a90-9d7366671883\") "
Sep 09 03:37:44.014170 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.014124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-plugins-config-volume\") pod \"cc83b8e1-ad9e-4cfd-8c3d-76a048762908\" (UID: \"cc83b8e1-ad9e-4cfd-8c3d-76a048762908\") "
Sep 09 03:37:44.014170 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.014145    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t5hz\" (UniqueName: \"kubernetes.io/projected/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-kube-api-access-8t5hz\") pod \"cc83b8e1-ad9e-4cfd-8c3d-76a048762908\" (UID: \"cc83b8e1-ad9e-4cfd-8c3d-76a048762908\") "
Sep 09 03:37:44.014256 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.014197    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wxsr\" (UniqueName: \"kubernetes.io/projected/a864a82a-6cbc-453e-9a90-9d7366671883-kube-api-access-6wxsr\") pod \"a864a82a-6cbc-453e-9a90-9d7366671883\" (UID: \"a864a82a-6cbc-453e-9a90-9d7366671883\") "
Sep 09 03:37:44.014504 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.014467    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc83b8e1-ad9e-4cfd-8c3d-76a048762908" (UID: "cc83b8e1-ad9e-4cfd-8c3d-76a048762908"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:44.014504 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.014469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a864a82a-6cbc-453e-9a90-9d7366671883-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a864a82a-6cbc-453e-9a90-9d7366671883" (UID: "a864a82a-6cbc-453e-9a90-9d7366671883"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:44.016386 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.016364    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a864a82a-6cbc-453e-9a90-9d7366671883-kube-api-access-6wxsr" (OuterVolumeSpecName: "kube-api-access-6wxsr") pod "a864a82a-6cbc-453e-9a90-9d7366671883" (UID: "a864a82a-6cbc-453e-9a90-9d7366671883"). InnerVolumeSpecName "kube-api-access-6wxsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:44.016446 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.016405    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-kube-api-access-8t5hz" (OuterVolumeSpecName: "kube-api-access-8t5hz") pod "cc83b8e1-ad9e-4cfd-8c3d-76a048762908" (UID: "cc83b8e1-ad9e-4cfd-8c3d-76a048762908"). InnerVolumeSpecName "kube-api-access-8t5hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:44.115619 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.115555    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wxsr\" (UniqueName: \"kubernetes.io/projected/a864a82a-6cbc-453e-9a90-9d7366671883-kube-api-access-6wxsr\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:37:44.115619 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.115613    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a864a82a-6cbc-453e-9a90-9d7366671883-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:37:44.115619 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.115623    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:37:44.115848 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.115632    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t5hz\" (UniqueName: \"kubernetes.io/projected/cc83b8e1-ad9e-4cfd-8c3d-76a048762908-kube-api-access-8t5hz\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:37:44.310442 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.310412    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"]
Sep 09 03:37:44.315179 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.315155    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cd1da6-7dd7d7b87d-kn699"]
Sep 09 03:37:44.324159 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.324135    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"]
Sep 09 03:37:44.326266 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.326244    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cd1da6-54d75bb487-r954h"]
Sep 09 03:37:44.970489 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.970450    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0325d57b-a494-4db4-ab84-32e2ad4308b2" path="/var/lib/kubelet/pods/0325d57b-a494-4db4-ab84-32e2ad4308b2/volumes"
Sep 09 03:37:44.970888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.970782    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e313e8-0f09-4809-b310-dcfde8b33915" path="/var/lib/kubelet/pods/83e313e8-0f09-4809-b310-dcfde8b33915/volumes"
Sep 09 03:37:44.971064 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.971051    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a864a82a-6cbc-453e-9a90-9d7366671883" path="/var/lib/kubelet/pods/a864a82a-6cbc-453e-9a90-9d7366671883/volumes"
Sep 09 03:37:44.971358 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:44.971347    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc83b8e1-ad9e-4cfd-8c3d-76a048762908" path="/var/lib/kubelet/pods/cc83b8e1-ad9e-4cfd-8c3d-76a048762908/volumes"
Sep 09 03:37:46.599151 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:46.599115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:37:56.003112 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:56.003079    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:37:57.203057 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:57.203020    2580 scope.go:117] "RemoveContainer" containerID="464eb423f12fdc2222a53924eb4abf5cc7e12895fddd6106839ca4de491d1067"
Sep 09 03:37:57.211638 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:57.211616    2580 scope.go:117] "RemoveContainer" containerID="216036402a4eab381c8d649ccefdca14ac9a1644413a2266271e696edfd4a803"
Sep 09 03:37:57.219687 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:37:57.219667    2580 scope.go:117] "RemoveContainer" containerID="3628f627109d638488d5903151efc964b97432017e99beaeda20b0f3a76321ad"
Sep 09 03:38:57.264146 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:38:57.264116    2580 scope.go:117] "RemoveContainer" containerID="275ddb4e788662a8022b29a30941ac068d98f57f1375c82ca5cb3a4331071740"
Sep 09 03:38:57.272382 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:38:57.272361    2580 scope.go:117] "RemoveContainer" containerID="b6d3881c45bd415493517d1c8e7ad7c0a2cae990993718b984dbb8e6cc2aeb11"
Sep 09 03:39:15.923070 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:39:15.923026    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:39:42.196763 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:39:42.196723    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:39:50.397299 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:39:50.397262    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:39:57.999685 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:39:57.999647    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:40:06.206701 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:40:06.206663    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:40:19.221888 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:40:19.221807    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:40:34.511298 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:40:34.511257    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:41:57.310186 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:41:57.310156    2580 scope.go:117] "RemoveContainer" containerID="3762440d70fb8c8bae7591bd53645591b5833ab6b84f45c35737208f993d7952"
Sep 09 03:41:57.318675 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:41:57.318653    2580 scope.go:117] "RemoveContainer" containerID="247193df10cb753044be457ff3b0e8cdd8cb6281e19c80866125161d3da11e49"
Sep 09 03:42:05.703428 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:05.703394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:42:14.398288 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:14.398248    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:42:21.702275 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:21.702243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:42:41.413101 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:41.413065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:42:50.707047 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:50.707012    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:42:58.831023 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.830983    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"]
Sep 09 03:42:58.831512 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.831355    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a864a82a-6cbc-453e-9a90-9d7366671883" containerName="payload-processing"
Sep 09 03:42:58.831512 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.831367    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a864a82a-6cbc-453e-9a90-9d7366671883" containerName="payload-processing"
Sep 09 03:42:58.831512 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.831377    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0325d57b-a494-4db4-ab84-32e2ad4308b2" containerName="revoke-keys"
Sep 09 03:42:58.831512 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.831384    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0325d57b-a494-4db4-ab84-32e2ad4308b2" containerName="revoke-keys"
Sep 09 03:42:58.831512 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.831400    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc83b8e1-ad9e-4cfd-8c3d-76a048762908" containerName="payload-pre-processing"
Sep 09 03:42:58.831512 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.831405    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc83b8e1-ad9e-4cfd-8c3d-76a048762908" containerName="payload-pre-processing"
Sep 09 03:42:58.831512 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.831459    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a864a82a-6cbc-453e-9a90-9d7366671883" containerName="payload-processing"
Sep 09 03:42:58.831512 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.831470    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc83b8e1-ad9e-4cfd-8c3d-76a048762908" containerName="payload-pre-processing"
Sep 09 03:42:58.831512 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.831477    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0325d57b-a494-4db4-ab84-32e2ad4308b2" containerName="revoke-keys"
Sep 09 03:42:58.834426 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.834408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.836967 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.836942    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:42:58.837924 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.837899    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 03:42:58.838053 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.837933    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 03:42:58.838053 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.837936    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 03:42:58.838177 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.838064    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6zwh9\""
Sep 09 03:42:58.848085 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.848064    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"]
Sep 09 03:42:58.873067 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.873038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5203d293-1631-40de-ba79-a193e55028af-openshift-service-ca\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.873241 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.873079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-metrics-certs\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.873241 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.873225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-maas-api-tls\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.873365 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.873258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z48sb\" (UniqueName: \"kubernetes.io/projected/5203d293-1631-40de-ba79-a193e55028af-kube-api-access-z48sb\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.973854 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.973824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-maas-api-tls\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.974059 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.973863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z48sb\" (UniqueName: \"kubernetes.io/projected/5203d293-1631-40de-ba79-a193e55028af-kube-api-access-z48sb\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.974059 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.973898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5203d293-1631-40de-ba79-a193e55028af-openshift-service-ca\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.974059 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.973932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-metrics-certs\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.974681 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.974658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5203d293-1631-40de-ba79-a193e55028af-openshift-service-ca\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.976510 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.976481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-metrics-certs\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.976657 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.976496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-maas-api-tls\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:58.981090 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:58.981062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z48sb\" (UniqueName: \"kubernetes.io/projected/5203d293-1631-40de-ba79-a193e55028af-kube-api-access-z48sb\") pod \"maas-api-65bcb85cfb-8s8hh\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") " pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:59.096940 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:59.096864    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:42:59.144421 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:59.144395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:42:59.416313 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:59.416278    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"]
Sep 09 03:42:59.419083 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:42:59.419038    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5203d293_1631_40de_ba79_a193e55028af.slice/crio-6ed1deeebf7424a26c544de8431cc7e706b6becd4fe90482555c91a8c29f513a WatchSource:0}: Error finding container 6ed1deeebf7424a26c544de8431cc7e706b6becd4fe90482555c91a8c29f513a: Status 404 returned error can't find the container with id 6ed1deeebf7424a26c544de8431cc7e706b6becd4fe90482555c91a8c29f513a
Sep 09 03:42:59.420343 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:59.420326    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:42:59.496322 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:42:59.496288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh" event={"ID":"5203d293-1631-40de-ba79-a193e55028af","Type":"ContainerStarted","Data":"6ed1deeebf7424a26c544de8431cc7e706b6becd4fe90482555c91a8c29f513a"}
Sep 09 03:43:01.508071 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:01.508029    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh" event={"ID":"5203d293-1631-40de-ba79-a193e55028af","Type":"ContainerStarted","Data":"7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455"}
Sep 09 03:43:01.508457 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:01.508177    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:43:01.524827 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:01.524769    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh" podStartSLOduration=1.589424758 podStartE2EDuration="3.524748359s" totalImagesPullingTime="1.935323601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:43:01.524114129 +0000 UTC m=+2225.050428870" watchObservedRunningTime="2026-09-09 03:43:01.524748359 +0000 UTC m=+2225.051063100"
Sep 09 03:43:06.125463 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.125428    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"]
Sep 09 03:43:06.126041 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.125710    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh" podUID="5203d293-1631-40de-ba79-a193e55028af" containerName="maas-api" containerID="cri-o://7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455" gracePeriod=30
Sep 09 03:43:06.131785 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.131762    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:43:06.157708 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.157675    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76695d568-zdr74"]
Sep 09 03:43:06.160818 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.160800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.172603 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.172546    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76695d568-zdr74"]
Sep 09 03:43:06.242869 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.242836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9825efdd-2185-4e6d-93e4-86428c57dd67-openshift-service-ca\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.243020 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.242883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9825efdd-2185-4e6d-93e4-86428c57dd67-maas-api-tls\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.243020 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.243000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9825efdd-2185-4e6d-93e4-86428c57dd67-metrics-certs\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.243100 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.243027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvxff\" (UniqueName: \"kubernetes.io/projected/9825efdd-2185-4e6d-93e4-86428c57dd67-kube-api-access-dvxff\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.343910 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.343879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9825efdd-2185-4e6d-93e4-86428c57dd67-metrics-certs\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.344094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.343918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvxff\" (UniqueName: \"kubernetes.io/projected/9825efdd-2185-4e6d-93e4-86428c57dd67-kube-api-access-dvxff\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.344094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.343945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9825efdd-2185-4e6d-93e4-86428c57dd67-openshift-service-ca\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.344094 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.343968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9825efdd-2185-4e6d-93e4-86428c57dd67-maas-api-tls\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.344569 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.344547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9825efdd-2185-4e6d-93e4-86428c57dd67-openshift-service-ca\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.346653 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.346630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9825efdd-2185-4e6d-93e4-86428c57dd67-maas-api-tls\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.346735 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.346630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9825efdd-2185-4e6d-93e4-86428c57dd67-metrics-certs\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.351588 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.351561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvxff\" (UniqueName: \"kubernetes.io/projected/9825efdd-2185-4e6d-93e4-86428c57dd67-kube-api-access-dvxff\") pod \"maas-api-76695d568-zdr74\" (UID: \"9825efdd-2185-4e6d-93e4-86428c57dd67\") " pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.377737 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.377688    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:43:06.445236 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.445194    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z48sb\" (UniqueName: \"kubernetes.io/projected/5203d293-1631-40de-ba79-a193e55028af-kube-api-access-z48sb\") pod \"5203d293-1631-40de-ba79-a193e55028af\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") "
Sep 09 03:43:06.445236 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.445242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-maas-api-tls\") pod \"5203d293-1631-40de-ba79-a193e55028af\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") "
Sep 09 03:43:06.445553 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.445320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5203d293-1631-40de-ba79-a193e55028af-openshift-service-ca\") pod \"5203d293-1631-40de-ba79-a193e55028af\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") "
Sep 09 03:43:06.445553 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.445444    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-metrics-certs\") pod \"5203d293-1631-40de-ba79-a193e55028af\" (UID: \"5203d293-1631-40de-ba79-a193e55028af\") "
Sep 09 03:43:06.445727 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.445703    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5203d293-1631-40de-ba79-a193e55028af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5203d293-1631-40de-ba79-a193e55028af" (UID: "5203d293-1631-40de-ba79-a193e55028af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:43:06.445844 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.445827    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5203d293-1631-40de-ba79-a193e55028af-openshift-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:43:06.447841 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.447815    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5203d293-1631-40de-ba79-a193e55028af" (UID: "5203d293-1631-40de-ba79-a193e55028af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:43:06.447965 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.447841    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5203d293-1631-40de-ba79-a193e55028af" (UID: "5203d293-1631-40de-ba79-a193e55028af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:43:06.447965 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.447842    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5203d293-1631-40de-ba79-a193e55028af-kube-api-access-z48sb" (OuterVolumeSpecName: "kube-api-access-z48sb") pod "5203d293-1631-40de-ba79-a193e55028af" (UID: "5203d293-1631-40de-ba79-a193e55028af"). InnerVolumeSpecName "kube-api-access-z48sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:43:06.501848 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.501805    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:43:06.522428 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.522396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:06.531445 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.531418    2580 generic.go:356] "Generic (PLEG): container finished" podID="5203d293-1631-40de-ba79-a193e55028af" containerID="7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455" exitCode=0
Sep 09 03:43:06.531554 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.531475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh" event={"ID":"5203d293-1631-40de-ba79-a193e55028af","Type":"ContainerDied","Data":"7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455"}
Sep 09 03:43:06.531554 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.531479    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"
Sep 09 03:43:06.531554 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.531522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh" event={"ID":"5203d293-1631-40de-ba79-a193e55028af","Type":"ContainerDied","Data":"6ed1deeebf7424a26c544de8431cc7e706b6becd4fe90482555c91a8c29f513a"}
Sep 09 03:43:06.531554 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.531548    2580 scope.go:117] "RemoveContainer" containerID="7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455"
Sep 09 03:43:06.541016 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.540996    2580 scope.go:117] "RemoveContainer" containerID="7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455"
Sep 09 03:43:06.541371 ip-10-0-138-152 kubenswrapper[2580]: E0909 03:43:06.541350    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455\": container with ID starting with 7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455 not found: ID does not exist" containerID="7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455"
Sep 09 03:43:06.541427 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.541382    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455"} err="failed to get container status \"7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455\": rpc error: code = NotFound desc = could not find container \"7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455\": container with ID starting with 7e6289a76cc9349c26ffb2413e1ea3183f47918570cc562a19d03d9f20139455 not found: ID does not exist"
Sep 09 03:43:06.547225 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.547198    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z48sb\" (UniqueName: \"kubernetes.io/projected/5203d293-1631-40de-ba79-a193e55028af-kube-api-access-z48sb\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:43:06.547225 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.547222    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-maas-api-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:43:06.547397 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.547231    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5203d293-1631-40de-ba79-a193e55028af-metrics-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Sep 09 03:43:06.553144 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.553117    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"]
Sep 09 03:43:06.561761 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.561705    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-65bcb85cfb-8s8hh"]
Sep 09 03:43:06.795897 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.795821    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76695d568-zdr74"]
Sep 09 03:43:06.799135 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:43:06.799109    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9825efdd_2185_4e6d_93e4_86428c57dd67.slice/crio-3b57692d3e73ea97b985f2870365c14588b7260c47377087fdef800deca1e8b4 WatchSource:0}: Error finding container 3b57692d3e73ea97b985f2870365c14588b7260c47377087fdef800deca1e8b4: Status 404 returned error can't find the container with id 3b57692d3e73ea97b985f2870365c14588b7260c47377087fdef800deca1e8b4
Sep 09 03:43:06.971087 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:06.971046    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5203d293-1631-40de-ba79-a193e55028af" path="/var/lib/kubelet/pods/5203d293-1631-40de-ba79-a193e55028af/volumes"
Sep 09 03:43:07.544110 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:07.544073    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74" event={"ID":"9825efdd-2185-4e6d-93e4-86428c57dd67","Type":"ContainerStarted","Data":"9be6653ad559dbdb76827ecba30591ae7cad8ca2bee397db4526d397ddef4ec2"}
Sep 09 03:43:07.544615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:07.544162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74" event={"ID":"9825efdd-2185-4e6d-93e4-86428c57dd67","Type":"ContainerStarted","Data":"3b57692d3e73ea97b985f2870365c14588b7260c47377087fdef800deca1e8b4"}
Sep 09 03:43:07.544615 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:07.544190    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:07.563263 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:07.563213    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74" podStartSLOduration=1.278420091 podStartE2EDuration="1.563199389s" totalImagesPullingTime="284.779298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:43:07.561880177 +0000 UTC m=+2231.088194917" watchObservedRunningTime="2026-09-09 03:43:07.563199389 +0000 UTC m=+2231.089514127"
Sep 09 03:43:13.555709 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:13.555678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76695d568-zdr74"
Sep 09 03:43:14.701872 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:14.701834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:43:22.403148 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:22.403071    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:43:29.703282 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:29.703241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fwzxn"]
Sep 09 03:43:39.640823 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:39.640790    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-g9jfv_fcd17b94-6090-4050-ad4e-c6045c1bf678/copy-manifests/0.log"
Sep 09 03:43:39.648009 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:39.647978    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-g9jfv_fcd17b94-6090-4050-ad4e-c6045c1bf678/manager/0.log"
Sep 09 03:43:40.267635 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:40.267597    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76695d568-zdr74_9825efdd-2185-4e6d-93e4-86428c57dd67/maas-api/0.log"
Sep 09 03:43:40.388886 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:40.388846    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29815410-qdrr5_0c2502ad-9a26-436f-8b5b-41dc0fd3a35d/cleanup/0.log"
Sep 09 03:43:41.952916 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:41.952890    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-fwzxn_f3247983-5d97-40d0-8535-6d666053f16c/limitador/0.log"
Sep 09 03:43:42.599626 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:42.599593    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-td6vb_7ff2e94c-a757-4c85-8473-0c7fe187f545/discovery/0.log"
Sep 09 03:43:43.077846 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:43.077816    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-69kbd_a9091a41-7a8b-4c6b-b561-66656c46e9c0/payload-pre-processing/0.log"
Sep 09 03:43:43.204410 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:43.204377    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kmv79_dcbb770c-adde-492a-92b4-22a3050ced12/payload-processing/0.log"
Sep 09 03:43:44.176479 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:44.176446    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpt9lt_00e2cb90-ef1e-4741-a44d-693be6c4f83b/main/0.log"
Sep 09 03:43:53.303062 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:53.303027    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-psdng_d8692cce-7d57-40d3-9994-7fdfc0adf178/global-pull-secret-syncer/0.log"
Sep 09 03:43:53.419758 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:53.419731    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z2zb6_4896cb96-b860-4f7c-aaa8-2abb525ed851/konnectivity-agent/0.log"
Sep 09 03:43:53.441872 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:53.441834    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-152.ec2.internal_9ce43aa27923e3198d26bf26fefbfe86/haproxy/0.log"
Sep 09 03:43:57.349608 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:57.349562    2580 scope.go:117] "RemoveContainer" containerID="fcf336b0c3de7ad53e9b4ddf4f362e8433a95ff8830405175a5f064ce38e4ee7"
Sep 09 03:43:57.358045 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:57.358021    2580 scope.go:117] "RemoveContainer" containerID="83e42f0b268871f8a4003ada631bc5758f601255bd475400bcf1e40dbc709685"
Sep 09 03:43:57.365854 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:57.365837    2580 scope.go:117] "RemoveContainer" containerID="c595e25429f9665799674a25ee585e7feba9f8a97c54c579700b6452483befe9"
Sep 09 03:43:58.588073 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:43:58.587994    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-fwzxn_f3247983-5d97-40d0-8535-6d666053f16c/limitador/0.log"
Sep 09 03:44:00.173650 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.173622    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae262f6-ff35-426b-a134-f891ad2bacd9/alertmanager/0.log"
Sep 09 03:44:00.200955 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.200929    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae262f6-ff35-426b-a134-f891ad2bacd9/config-reloader/0.log"
Sep 09 03:44:00.222179 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.222155    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae262f6-ff35-426b-a134-f891ad2bacd9/kube-rbac-proxy-web/0.log"
Sep 09 03:44:00.242636 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.242600    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae262f6-ff35-426b-a134-f891ad2bacd9/kube-rbac-proxy/0.log"
Sep 09 03:44:00.264666 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.264639    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae262f6-ff35-426b-a134-f891ad2bacd9/kube-rbac-proxy-metric/0.log"
Sep 09 03:44:00.288000 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.287967    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae262f6-ff35-426b-a134-f891ad2bacd9/prom-label-proxy/0.log"
Sep 09 03:44:00.308951 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.308911    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae262f6-ff35-426b-a134-f891ad2bacd9/init-config-reloader/0.log"
Sep 09 03:44:00.369256 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.369222    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-c8nnl_1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f/kube-state-metrics/0.log"
Sep 09 03:44:00.386597 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.386559    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-c8nnl_1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f/kube-rbac-proxy-main/0.log"
Sep 09 03:44:00.405483 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.405455    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-c8nnl_1bf3f4ef-16d7-4ca7-bf47-6991a72b7c6f/kube-rbac-proxy-self/0.log"
Sep 09 03:44:00.435863 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.435782    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-58bd7d44c7-7fm62_1029aa7c-77fc-485c-9b25-e048b206a67f/metrics-server/0.log"
Sep 09 03:44:00.581235 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.581209    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgfkv_482d3f71-6af7-49ac-8483-04240b51ae4d/node-exporter/0.log"
Sep 09 03:44:00.604860 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.604825    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgfkv_482d3f71-6af7-49ac-8483-04240b51ae4d/kube-rbac-proxy/0.log"
Sep 09 03:44:00.628634 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.628606    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgfkv_482d3f71-6af7-49ac-8483-04240b51ae4d/init-textfile/0.log"
Sep 09 03:44:00.904151 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.904120    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d/prometheus/0.log"
Sep 09 03:44:00.923759 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.923714    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d/config-reloader/0.log"
Sep 09 03:44:00.946051 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.946021    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d/thanos-sidecar/0.log"
Sep 09 03:44:00.971097 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.971060    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d/kube-rbac-proxy-web/0.log"
Sep 09 03:44:00.994959 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:00.994926    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d/kube-rbac-proxy/0.log"
Sep 09 03:44:01.023531 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:01.023450    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d/kube-rbac-proxy-thanos/0.log"
Sep 09 03:44:01.046525 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:01.046497    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec5e74ad-e8a7-4064-8b5b-f5eb1f77633d/init-config-reloader/0.log"
Sep 09 03:44:01.075856 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:01.075805    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-cbqdd_451934bb-acab-4369-9512-200198b619cf/prometheus-operator/0.log"
Sep 09 03:44:01.094176 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:01.094156    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-cbqdd_451934bb-acab-4369-9512-200198b619cf/kube-rbac-proxy/0.log"
Sep 09 03:44:03.934226 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:03.934181    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"]
Sep 09 03:44:03.934773 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:03.934538    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5203d293-1631-40de-ba79-a193e55028af" containerName="maas-api"
Sep 09 03:44:03.934773 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:03.934550    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5203d293-1631-40de-ba79-a193e55028af" containerName="maas-api"
Sep 09 03:44:03.934773 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:03.934639    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5203d293-1631-40de-ba79-a193e55028af" containerName="maas-api"
Sep 09 03:44:03.937646 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:03.937630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:03.939845 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:03.939814    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nn8m\"/\"kube-root-ca.crt\""
Sep 09 03:44:03.940003 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:03.939909    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nn8m\"/\"openshift-service-ca.crt\""
Sep 09 03:44:03.940003 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:03.939978    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9nn8m\"/\"default-dockercfg-k9ktg\""
Sep 09 03:44:03.945351 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:03.945331    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"]
Sep 09 03:44:04.069029 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.068986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-proc\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.069251 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.069078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-sys\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.069251 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.069104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-lib-modules\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.069251 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.069126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlszh\" (UniqueName: \"kubernetes.io/projected/1ca73113-e861-4162-b87c-820f1e77fb36-kube-api-access-wlszh\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.069251 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.069218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-podres\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.170771 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.170724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-podres\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.170984 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.170793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-proc\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.170984 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.170837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-sys\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.170984 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.170864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-lib-modules\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.170984 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.170889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlszh\" (UniqueName: \"kubernetes.io/projected/1ca73113-e861-4162-b87c-820f1e77fb36-kube-api-access-wlszh\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.170984 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.170921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-podres\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.170984 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.170932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-proc\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.170984 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.170939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-sys\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.171236 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.171020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ca73113-e861-4162-b87c-820f1e77fb36-lib-modules\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.179770 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.179737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlszh\" (UniqueName: \"kubernetes.io/projected/1ca73113-e861-4162-b87c-820f1e77fb36-kube-api-access-wlszh\") pod \"perf-node-gather-daemonset-m544d\" (UID: \"1ca73113-e861-4162-b87c-820f1e77fb36\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.249116 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.249016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.519344 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.519317    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"]
Sep 09 03:44:04.521836 ip-10-0-138-152 kubenswrapper[2580]: W0909 03:44:04.521803    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1ca73113_e861_4162_b87c_820f1e77fb36.slice/crio-0a38c7f6d5117b5e52b3bac794fc6817a0f3b3dbbdcc8d1966b9cc38dac30c60 WatchSource:0}: Error finding container 0a38c7f6d5117b5e52b3bac794fc6817a0f3b3dbbdcc8d1966b9cc38dac30c60: Status 404 returned error can't find the container with id 0a38c7f6d5117b5e52b3bac794fc6817a0f3b3dbbdcc8d1966b9cc38dac30c60
Sep 09 03:44:04.813441 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.813339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d" event={"ID":"1ca73113-e861-4162-b87c-820f1e77fb36","Type":"ContainerStarted","Data":"0e092cd90a0d08c61bfc34a007db95fa46c4ccd5f3fe6f8b6eea42f6ac1b1f7c"}
Sep 09 03:44:04.813441 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.813427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d" event={"ID":"1ca73113-e861-4162-b87c-820f1e77fb36","Type":"ContainerStarted","Data":"0a38c7f6d5117b5e52b3bac794fc6817a0f3b3dbbdcc8d1966b9cc38dac30c60"}
Sep 09 03:44:04.813719 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.813483    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:04.830022 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:04.829973    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d" podStartSLOduration=1.8299576119999998 podStartE2EDuration="1.829957612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:44:04.827806885 +0000 UTC m=+2288.354121625" watchObservedRunningTime="2026-09-09 03:44:04.829957612 +0000 UTC m=+2288.356272350"
Sep 09 03:44:05.857189 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:05.857159    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mn566_3ed10abb-1e74-433a-8d98-c07d0d7d99ff/dns/0.log"
Sep 09 03:44:05.899717 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:05.899691    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mn566_3ed10abb-1e74-433a-8d98-c07d0d7d99ff/kube-rbac-proxy/0.log"
Sep 09 03:44:06.025378 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:06.025349    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s6hwl_0d29b378-9477-47b3-9500-c47d907fbe49/dns-node-resolver/0.log"
Sep 09 03:44:06.724221 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:06.724185    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jwqmr_f4039f93-f06b-4c04-9870-ff92b01a66f4/node-ca/0.log"
Sep 09 03:44:07.915415 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:07.915387    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-td6vb_7ff2e94c-a757-4c85-8473-0c7fe187f545/discovery/0.log"
Sep 09 03:44:08.044348 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:08.044324    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-69kbd_a9091a41-7a8b-4c6b-b561-66656c46e9c0/payload-pre-processing/0.log"
Sep 09 03:44:08.077280 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:08.077249    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kmv79_dcbb770c-adde-492a-92b4-22a3050ced12/payload-processing/0.log"
Sep 09 03:44:09.019927 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:09.019896    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nm9ch_4fa0b01d-e400-4fc5-b7b9-12032ba010ca/serve-healthcheck-canary/0.log"
Sep 09 03:44:09.589630 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:09.589597    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-46d6q_d5e2ce87-6721-4f8b-9daa-dcbaea830480/kube-rbac-proxy/0.log"
Sep 09 03:44:09.608908 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:09.608886    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-46d6q_d5e2ce87-6721-4f8b-9daa-dcbaea830480/exporter/0.log"
Sep 09 03:44:09.628546 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:09.628517    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-46d6q_d5e2ce87-6721-4f8b-9daa-dcbaea830480/extractor/0.log"
Sep 09 03:44:10.828628 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:10.828598    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-m544d"
Sep 09 03:44:12.330905 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:12.330874    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-g9jfv_fcd17b94-6090-4050-ad4e-c6045c1bf678/manager/0.log"
Sep 09 03:44:12.361711 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:12.361688    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-g9jfv_fcd17b94-6090-4050-ad4e-c6045c1bf678/copy-manifests/0.log"
Sep 09 03:44:13.900652 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:13.900618    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f6f9bc9b9-9n89j_25c05025-244b-45ff-a9e6-d7123cbde3e4/manager/0.log"
Sep 09 03:44:15.135472 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:15.135437    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-td6vb_7ff2e94c-a757-4c85-8473-0c7fe187f545/discovery/0.log"
Sep 09 03:44:15.217976 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:15.217940    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-69kbd_a9091a41-7a8b-4c6b-b561-66656c46e9c0/payload-pre-processing/0.log"
Sep 09 03:44:15.245931 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:15.245905    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kmv79_dcbb770c-adde-492a-92b4-22a3050ced12/payload-processing/0.log"
Sep 09 03:44:17.178760 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:17.178725    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-td6vb_7ff2e94c-a757-4c85-8473-0c7fe187f545/discovery/0.log"
Sep 09 03:44:17.282634 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:17.282602    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-69kbd_a9091a41-7a8b-4c6b-b561-66656c46e9c0/payload-pre-processing/0.log"
Sep 09 03:44:17.305782 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:17.305753    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kmv79_dcbb770c-adde-492a-92b4-22a3050ced12/payload-processing/0.log"
Sep 09 03:44:20.637368 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:20.637341    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72j6p_a8ab8583-facb-4d47-9197-83089e5468b0/kube-multus-additional-cni-plugins/0.log"
Sep 09 03:44:20.659096 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:20.659066    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72j6p_a8ab8583-facb-4d47-9197-83089e5468b0/egress-router-binary-copy/0.log"
Sep 09 03:44:20.679792 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:20.679762    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72j6p_a8ab8583-facb-4d47-9197-83089e5468b0/cni-plugins/0.log"
Sep 09 03:44:20.700974 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:20.700947    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72j6p_a8ab8583-facb-4d47-9197-83089e5468b0/bond-cni-plugin/0.log"
Sep 09 03:44:20.730356 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:20.725428    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72j6p_a8ab8583-facb-4d47-9197-83089e5468b0/routeoverride-cni/0.log"
Sep 09 03:44:20.748079 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:20.748059    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72j6p_a8ab8583-facb-4d47-9197-83089e5468b0/whereabouts-cni-bincopy/0.log"
Sep 09 03:44:20.766703 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:20.766678    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72j6p_a8ab8583-facb-4d47-9197-83089e5468b0/whereabouts-cni/0.log"
Sep 09 03:44:21.166471 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:21.166435    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cd7wf_ee670977-001f-4a64-8c78-4245ad3befe8/kube-multus/0.log"
Sep 09 03:44:21.238162 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:21.238118    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6kfmb_6d835db3-17d5-457e-a997-a2d7a61b516d/network-metrics-daemon/0.log"
Sep 09 03:44:21.255161 ip-10-0-138-152 kubenswrapper[2580]: I0909 03:44:21.255132    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6kfmb_6d835db3-17d5-457e-a997-a2d7a61b516d/kube-rbac-proxy/0.log"