Aug 04 20:40:25.123616 ip-10-0-140-103 systemd[1]: Starting Kubernetes Kubelet...
Aug 04 20:40:25.559510 ip-10-0-140-103 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 20:40:25.559510 ip-10-0-140-103 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 04 20:40:25.559510 ip-10-0-140-103 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 20:40:25.559510 ip-10-0-140-103 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 04 20:40:25.559510 ip-10-0-140-103 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 20:40:25.562335 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.562253    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 04 20:40:25.569851 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569832    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 20:40:25.569851 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569849    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 20:40:25.569851 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569852    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 20:40:25.569938 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569855    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 20:40:25.569938 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569858    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 20:40:25.569938 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569861    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 20:40:25.569938 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569864    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 20:40:25.569938 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569916    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569971    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569976    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569983    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569988    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569992    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569995    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.569999    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570001    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570005    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570008    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570011    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570014    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570020    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570023    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570028    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570034    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570039    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570043    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570048    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 20:40:25.570075 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.570052    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 20:40:25.571285 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571262    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 20:40:25.571395 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571386    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571396    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571402    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571406    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571409    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571413    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571415    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571419    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571422    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571424    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571428    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571430    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571433    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 20:40:25.571430 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571436    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571439    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571442    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571445    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571449    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571452    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571454    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571457    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571460    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571462    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571465    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571468    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571472    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571479    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571482    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571485    2565 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571488    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571491    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571494    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 20:40:25.571789 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571497    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571500    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571504    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571506    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571509    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571511    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571514    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571517    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571522    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571525    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571527    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571530    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571533    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571536    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571538    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571541    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571544    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571547    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571550    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571553    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 20:40:25.572251 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571556    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571558    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571561    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571564    2565 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571566    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571569    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571572    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571575    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571578    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571581    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571583    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571586    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571589    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571591    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571594    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571597    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571599    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.571602    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572314    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572321    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 20:40:25.572791 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572324    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572327    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572330    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572333    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572335    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572338    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572359    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572362    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572365    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572367    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572370    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572372    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572375    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572377    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572382    2565 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572386    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572390    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572393    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572395    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572398    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 20:40:25.573267 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572401    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572403    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572407    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572410    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572412    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572415    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572418    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572420    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572423    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572425    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572428    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572430    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572433    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572435    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572438    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572440    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572443    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572446    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572448    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572451    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 20:40:25.573862 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572453    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572456    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572459    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572461    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572464    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572466    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572470    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572472    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572475    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572478    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572481    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572483    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572486    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572488    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572491    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572494    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572497    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572500    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572502    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572505    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 20:40:25.574384 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572507    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572510    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572514    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572518    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572521    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572524    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572527    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572530    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572533    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572535    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572539    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572542    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572544    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572547    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572550    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572552    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572555    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572557    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572560    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 20:40:25.574869 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572563    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572567    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572570    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572573    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572576    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572579    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572582    2565 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572584    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572588    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572591    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572594    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572597    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572600    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572602    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572605    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572608    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572611    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572613    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572990    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 20:40:25.575336 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.572995    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573000    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573004    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573007    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573011    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573014    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573017    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573020    2565 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573022    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573025    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573028    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573031    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573034    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573036    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573039    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573042    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573044    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573047    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573050    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 20:40:25.575837 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573052    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573055    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573057    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573061    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573064    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573066    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573069    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573072    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573078    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573080    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573083    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573086    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573088    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573091    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573093    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573096    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573099    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573101    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573104    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573106    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 20:40:25.576305 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573109    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573111    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573114    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573116    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573119    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573122    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573124    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573127    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573130    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573134    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573137    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573141    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573143    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573146    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573149    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573152    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573155    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573158    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573161    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573163    2565 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 20:40:25.576821 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573167    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573170    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573173    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573175    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573178    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573181    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573183    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573187    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573189    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573192    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573194    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573197    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573199    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573202    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573205    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573207    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573210    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573215    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573218    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573220    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 20:40:25.577356 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573222    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573225    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573229    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573232    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573235    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573237    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573240    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573243    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573245    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573249    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573251    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573254    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573259    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573262    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573265    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573267    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573270    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573272    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573275    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573360    2565 flags.go:64] FLAG: --address="0.0.0.0"
Aug 04 20:40:25.577852 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573368    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573374    2565 flags.go:64] FLAG: --anonymous-auth="true"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573378    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573382    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573385    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573390    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573394    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573398    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573401    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573406    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573410    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573413    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573417    2565 flags.go:64] FLAG: --cgroup-root=""
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573420    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573423    2565 flags.go:64] FLAG: --client-ca-file=""
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573425    2565 flags.go:64] FLAG: --cloud-provider="external"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573428    2565 flags.go:64] FLAG: --cluster-dns="[]"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573434    2565 flags.go:64] FLAG: --cluster-domain=""
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573437    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573440    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573443    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573446    2565 flags.go:64] FLAG: --container-log-max-files="5"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573451    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 04 20:40:25.578392 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573454    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573458    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573461    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573464    2565 flags.go:64] FLAG: --contention-profiling="false"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573467    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573470    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573473    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573476    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573480    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573483    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573486    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573489    2565 flags.go:64] FLAG: --enable-load-reader="false"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573492    2565 flags.go:64] FLAG: --enable-server="true"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573495    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573499    2565 flags.go:64] FLAG: --event-burst="100"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573502    2565 flags.go:64] FLAG: --event-qps="50"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573505    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573508    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573512    2565 flags.go:64] FLAG: --eviction-hard=""
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573516    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573519    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573522    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573525    2565 flags.go:64] FLAG: --eviction-soft=""
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573528    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573531    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 04 20:40:25.578971 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573534    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573537    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573540    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573542    2565 flags.go:64] FLAG: --fail-swap-on="true"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573545    2565 flags.go:64] FLAG: --feature-gates=""
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573552    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573555    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573559    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573563    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573566    2565 flags.go:64] FLAG: --healthz-port="10248"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573569    2565 flags.go:64] FLAG: --help="false"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573572    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573575    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573577    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573580    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573584    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573587    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573590    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573593    2565 flags.go:64] FLAG: --image-service-endpoint=""
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573596    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573599    2565 flags.go:64] FLAG: --kube-api-burst="100"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573602    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573605    2565 flags.go:64] FLAG: --kube-api-qps="50"
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573608    2565 flags.go:64] FLAG: --kube-reserved=""
Aug 04 20:40:25.579625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573610    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573614    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573617    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573620    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573623    2565 flags.go:64] FLAG: --lock-file=""
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573626    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573629    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573632    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573637    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573640    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573643    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573646    2565 flags.go:64] FLAG: --logging-format="text"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573649    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573652    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573655    2565 flags.go:64] FLAG: --manifest-url=""
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573658    2565 flags.go:64] FLAG: --manifest-url-header=""
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573663    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573666    2565 flags.go:64] FLAG: --max-open-files="1000000"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573670    2565 flags.go:64] FLAG: --max-pods="110"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573673    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573676    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573679    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573682    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573685    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573688    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 04 20:40:25.580216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573691    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573698    2565 flags.go:64] FLAG: --node-status-max-images="50"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573701    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573704    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573707    2565 flags.go:64] FLAG: --pod-cidr=""
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573710    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573715    2565 flags.go:64] FLAG: --pod-manifest-path=""
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573718    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573721    2565 flags.go:64] FLAG: --pods-per-core="0"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573725    2565 flags.go:64] FLAG: --port="10250"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573728    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573731    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e6bc6ee9372b81a2"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573734    2565 flags.go:64] FLAG: --qos-reserved=""
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573737    2565 flags.go:64] FLAG: --read-only-port="10255"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573740    2565 flags.go:64] FLAG: --register-node="true"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573743    2565 flags.go:64] FLAG: --register-with-taints=""
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573747    2565 flags.go:64] FLAG: --registry-burst="10"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573750    2565 flags.go:64] FLAG: --registry-qps="5"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573753    2565 flags.go:64] FLAG: --reserved-cpus=""
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573755    2565 flags.go:64] FLAG: --reserved-memory=""
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573759    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573762    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573766    2565 flags.go:64] FLAG: --rotate-certificates="false"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573769    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573772    2565 flags.go:64] FLAG: --runonce="false"
Aug 04 20:40:25.580890 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573775    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573778    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573781    2565 flags.go:64] FLAG: --seccomp-default="false"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573785    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573788    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573791    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573793    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573796    2565 flags.go:64] FLAG: --storage-driver-password="root"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573799    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573802    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573805    2565 flags.go:64] FLAG: --storage-driver-user="root"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573808    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573811    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573814    2565 flags.go:64] FLAG: --system-cgroups=""
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573817    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573822    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573826    2565 flags.go:64] FLAG: --tls-cert-file=""
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573829    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573833    2565 flags.go:64] FLAG: --tls-min-version=""
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573836    2565 flags.go:64] FLAG: --tls-private-key-file=""
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573839    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573842    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573844    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573847    2565 flags.go:64] FLAG: --v="2"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573851    2565 flags.go:64] FLAG: --version="false"
Aug 04 20:40:25.581516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573855    2565 flags.go:64] FLAG: --vmodule=""
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573859    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.573862    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573974    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573978    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573983    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573986    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573989    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573992    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573994    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.573999    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574002    2565 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574005    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574008    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574011    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574014    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574018    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574022    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574025    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 20:40:25.582143 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574028    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574030    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574033    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574036    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574040    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574042    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574046    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574048    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574051    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574054    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574056    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574059    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574061    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574064    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574066    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574069    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574071    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574074    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574078    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574080    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 20:40:25.582625 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574083    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574086    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574089    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574092    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574095    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574098    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574100    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574103    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574105    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574108    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574110    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574114    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574116    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574119    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574121    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574124    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574127    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574130    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574133    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574136    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 20:40:25.583114 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574138    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574141    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574143    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574146    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574148    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574151    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574153    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574157    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574161    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574165    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574168    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574171    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574174    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574176    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574179    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574182    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574185    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574187    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574190    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 20:40:25.583699 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574192    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574195    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574197    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574200    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574202    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574205    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574207    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574210    2565 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574212    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574216    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574219    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574221    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574224    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574227    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574229    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574232    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574234    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574237    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574239    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574242    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 20:40:25.584164 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574244    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574247    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574250    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.574253    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.574949    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.581987    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.582003    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.582022    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.582031    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582086    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582091    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582094    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582097    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582101    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582104    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 20:40:25.584677 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582107    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582110    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582113    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582116    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582118    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582121    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582124    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582127    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582130    2565 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582132    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582135    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582138    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582140    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582143    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582146    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582149    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582153    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582157    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582160    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 20:40:25.585052 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582163    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582165    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582169    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582171    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582175    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582178    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582181    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582183    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582186    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582188    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582191    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582194    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582196    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582199    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582203    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582206    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582209    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582211    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582214    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 20:40:25.585526 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582216    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582219    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582222    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582224    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582227    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582229    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582232    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582235    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582238    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582241    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582243    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582246    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582248    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582251    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582253    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582256    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582259    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582262    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582266    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582268    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 20:40:25.585985 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582272    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582275    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582278    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582281    2565 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582283    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582286    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582288    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582291    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582293    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582296    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582299    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582301    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582304    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582306    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582308    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582311    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582314    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582316    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582319    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582321    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 20:40:25.586519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582324    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582327    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582329    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582332    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582335    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582337    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582354    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582356    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582359    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582362    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582365    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582368    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582371    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582374    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582376    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 20:40:25.587003 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.582382    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582512    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582517    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582520    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582523    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582526    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582528    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582531    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582534    2565 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582537    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582540    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582542    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582545    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582547    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582550    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582554    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582559    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582562    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582564    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 20:40:25.587397 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582567    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582569    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582572    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582574    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582577    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582580    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582582    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582585    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582587    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582590    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582593    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582596    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582599    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582602    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582604    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582607    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582609    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582612    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582614    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 20:40:25.587863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582617    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582620    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582623    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582625    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582629    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582631    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582634    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582637    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582639    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582642    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582645    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582648    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582650    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582652    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582655    2565 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582658    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582660    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582662    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582667    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582670    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 20:40:25.588321 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582672    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582675    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582678    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582681    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582684    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582687    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582690    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582693    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582695    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582698    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582701    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582703    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582706    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582709    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582712    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582715    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582717    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582720    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582723    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582725    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 20:40:25.588834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582728    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582731    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582733    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582736    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582738    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582741    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582744    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582746    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582749    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582751    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582754    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582756    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582759    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582762    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582765    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582768    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582772    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582775    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582778    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582781    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 20:40:25.589313 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582783    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 20:40:25.589813 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:25.582786    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 20:40:25.589813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.582790    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 20:40:25.589813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.582892    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 04 20:40:25.589813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.586116    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 04 20:40:25.589813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.587871    2565 server.go:1019] "Starting client certificate rotation"
Aug 04 20:40:25.589813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.587965    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 20:40:25.589813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.587994    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 20:40:25.590683 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.590670    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 20:40:25.591819 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.591803    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 20:40:25.603354 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.603317    2565 log.go:25] "Validated CRI v1 runtime API"
Aug 04 20:40:25.605726 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.605711    2565 log.go:25] "Validated CRI v1 image API"
Aug 04 20:40:25.608679 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.608662    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 04 20:40:25.615443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.615420    2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a7b2f12f-2105-46d1-a435-9894ed1d0756:/dev/nvme0n1p3 c0d181bf-afbf-4e16-985a-71cef69f7af2:/dev/nvme0n1p4]
Aug 04 20:40:25.615529 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.615441    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 04 20:40:25.620717 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.620600    2565 manager.go:216] Machine: {Timestamp:2026-08-04 20:40:25.619165734 +0000 UTC m=+0.371271703 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100541 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec291ea354807dfa661663f472174dfb SystemUUID:ec291ea3-5480-7dfa-6616-63f472174dfb BootID:da07f597-5217-4e41-90a5-621dfe07b1b1 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:61:67:3d:73 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:61:67:3d:73 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:5c:24:89:a3:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 04 20:40:25.620717 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.620710    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 04 20:40:25.620845 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.620786    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 04 20:40:25.624214 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.624190    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 04 20:40:25.624389 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.624218    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-103.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 04 20:40:25.625094 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.625082    2565 topology_manager.go:138] "Creating topology manager with none policy"
Aug 04 20:40:25.625129 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.625098    2565 container_manager_linux.go:309] "Creating device plugin manager"
Aug 04 20:40:25.625129 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.625108    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 20:40:25.625935 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.625924    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 20:40:25.627847 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.627837    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 04 20:40:25.628808 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.628793    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 20:40:25.633122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.633111    2565 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 20:40:25.633236 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.633227    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 04 20:40:25.633264 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.633258    2565 kubelet.go:486] "Attempting to sync node with API server"
Aug 04 20:40:25.633296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.633272    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 04 20:40:25.633296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.633292    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 04 20:40:25.633364 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.633305    2565 kubelet.go:398] "Adding apiserver pod source"
Aug 04 20:40:25.633364 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.633317    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 04 20:40:25.633737 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.633726    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 20:40:25.633774 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.633751    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 20:40:25.636066 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.636051    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 04 20:40:25.637981 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.637966    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 04 20:40:25.638069 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638003    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 04 20:40:25.638796 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638782    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 04 20:40:25.638894 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638804    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 04 20:40:25.638894 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638813    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 04 20:40:25.638894 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638821    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 04 20:40:25.638894 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638832    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 04 20:40:25.638894 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638841    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 04 20:40:25.638894 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638853    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 04 20:40:25.638894 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638861    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 04 20:40:25.638894 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638870    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 04 20:40:25.638894 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638879    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 04 20:40:25.638894 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638897    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 04 20:40:25.639176 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.638909    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 04 20:40:25.639793 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.639780    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 04 20:40:25.639854 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.639795    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 04 20:40:25.644824 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.644802    2565 server.go:1268] "Started kubelet"
Aug 04 20:40:25.644922 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.644869    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 04 20:40:25.645529 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.645114    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 04 20:40:25.645529 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.645186    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 04 20:40:25.645659 ip-10-0-140-103 systemd[1]: Started Kubernetes Kubelet.
Aug 04 20:40:25.645892 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.645801    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 04 20:40:25.646658 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.646643    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 04 20:40:25.646963 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.646951    2565 server.go:310] "Adding debug handlers to kubelet server"
Aug 04 20:40:25.650955 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.650929    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 04 20:40:25.653934 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.653908    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-103.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 04 20:40:25.654004 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.653960    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 04 20:40:25.654994 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.653827    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-103.ec2.internal.18c8b4c8896d8dae  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-103.ec2.internal,UID:ip-10-0-140-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-103.ec2.internal,},FirstTimestamp:2026-08-04 20:40:25.64451883 +0000 UTC m=+0.396624799,LastTimestamp:2026-08-04 20:40:25.64451883 +0000 UTC m=+0.396624799,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-103.ec2.internal,}"
Aug 04 20:40:25.657745 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.657723    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 04 20:40:25.657745 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.657729    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 04 20:40:25.658016 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.657990    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:25.658567 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.658548    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 04 20:40:25.658567 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.658549    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 04 20:40:25.658703 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.658580    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 04 20:40:25.658703 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.658662    2565 reconstruct.go:97] "Volume reconstruction finished"
Aug 04 20:40:25.658703 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.658674    2565 reconciler.go:29] "Reconciler: start to sync state"
Aug 04 20:40:25.659131 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.659114    2565 factory.go:153] Registering CRI-O factory
Aug 04 20:40:25.659131 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.659133    2565 factory.go:223] Registration of the crio container factory successfully
Aug 04 20:40:25.659242 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.659193    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 04 20:40:25.659242 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.659205    2565 factory.go:55] Registering systemd factory
Aug 04 20:40:25.659242 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.659211    2565 factory.go:223] Registration of the systemd container factory successfully
Aug 04 20:40:25.659242 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.659232    2565 factory.go:103] Registering Raw factory
Aug 04 20:40:25.659427 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.659246    2565 manager.go:1196] Started watching for new ooms in manager
Aug 04 20:40:25.661259 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.660983    2565 manager.go:349] Starting recovery of all containers
Aug 04 20:40:25.661367 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.661323    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 04 20:40:25.661367 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.661337    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 04 20:40:25.673784 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.673762    2565 manager.go:354] Recovery completed
Aug 04 20:40:25.679454 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.679438    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 20:40:25.681360 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.681330    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 20:40:25.681424 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.681375    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 20:40:25.681424 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.681389    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID"
Aug 04 20:40:25.682031 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.682016    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 04 20:40:25.682031 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.682025    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 20:40:25.682031 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.682032    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 04 20:40:25.682222 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.682053    2565 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 20:40:25.682793 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.682778    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 20:40:25.682855 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.682805    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 20:40:25.682855 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.682819    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID"
Aug 04 20:40:25.684164 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.684149    2565 policy_none.go:49] "None policy: Start"
Aug 04 20:40:25.684164 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.684166    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 04 20:40:25.684267 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.684177    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 04 20:40:25.686200 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.686187    2565 policy_none.go:47] "Start"
Aug 04 20:40:25.692964 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.692894    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-103.ec2.internal.18c8b4c88b9fb159  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-103.ec2.internal,UID:ip-10-0-140-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-103.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-103.ec2.internal,},FirstTimestamp:2026-08-04 20:40:25.681359193 +0000 UTC m=+0.433465165,LastTimestamp:2026-08-04 20:40:25.681359193 +0000 UTC m=+0.433465165,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-103.ec2.internal,}"
Aug 04 20:40:25.702076 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.702011    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-103.ec2.internal.18c8b4c88ba006e3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-103.ec2.internal,UID:ip-10-0-140-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-103.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-103.ec2.internal,},FirstTimestamp:2026-08-04 20:40:25.681381091 +0000 UTC m=+0.433487060,LastTimestamp:2026-08-04 20:40:25.681381091 +0000 UTC m=+0.433487060,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-103.ec2.internal,}"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.709824    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xp2kg"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.711606    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-103.ec2.internal.18c8b4c88ba03b03  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-103.ec2.internal,UID:ip-10-0-140-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-103.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-103.ec2.internal,},FirstTimestamp:2026-08-04 20:40:25.681394435 +0000 UTC m=+0.433500404,LastTimestamp:2026-08-04 20:40:25.681394435 +0000 UTC m=+0.433500404,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-103.ec2.internal,}"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.718327    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xp2kg"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.722768    2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-103.ec2.internal.18c8b4c88b9fb159\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-103.ec2.internal.18c8b4c88b9fb159  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-103.ec2.internal,UID:ip-10-0-140-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-103.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-103.ec2.internal,},FirstTimestamp:2026-08-04 20:40:25.681359193 +0000 UTC m=+0.433465165,LastTimestamp:2026-08-04 20:40:25.682791686 +0000 UTC m=+0.434897656,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-103.ec2.internal,}"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.724925    2565 manager.go:337] "Starting Device Plugin manager"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.724947    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.724956    2565 server.go:87] "Starting device plugin registration server"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.725179    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.725193    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.725292    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.725395    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.725408    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.725815    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 04 20:40:25.729961 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.725941    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:25.752723 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.752687    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 04 20:40:25.753891 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.753874    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 04 20:40:25.754000 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.753912    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 04 20:40:25.754000 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.753930    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 04 20:40:25.754000 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.753960    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 04 20:40:25.756129 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.756110    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 20:40:25.826088 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.826037    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 20:40:25.827219 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.827198    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 20:40:25.827307 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.827230    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 20:40:25.827307 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.827245    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID"
Aug 04 20:40:25.827307 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.827267    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.841723 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.841707    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.841723 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.841724    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-103.ec2.internal\": node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:25.854987 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.854966    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal"]
Aug 04 20:40:25.855062 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.855022    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 20:40:25.855785 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.855771    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 20:40:25.855853 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.855797    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 20:40:25.855853 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.855811    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID"
Aug 04 20:40:25.858191 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.858177    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 20:40:25.858307 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.858294    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.858376 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.858328    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 20:40:25.858978 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.858964    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 20:40:25.859046 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.858977    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 20:40:25.859046 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.858987    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 20:40:25.859046 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.858997    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID"
Aug 04 20:40:25.859046 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.858999    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 20:40:25.859046 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.859009    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID"
Aug 04 20:40:25.859745 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.859729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b05ec092a95b70daec77eec1e81aa44-config\") pod \"kube-apiserver-proxy-ip-10-0-140-103.ec2.internal\" (UID: \"8b05ec092a95b70daec77eec1e81aa44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.859814 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.859752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/156f995d1a4076879b985132336f763d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"156f995d1a4076879b985132336f763d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.859814 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.859777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/156f995d1a4076879b985132336f763d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"156f995d1a4076879b985132336f763d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.861219 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.861205    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.861269 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.861238    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 20:40:25.861920 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.861905    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 20:40:25.861985 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.861934    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 20:40:25.861985 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.861947    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID"
Aug 04 20:40:25.863040 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.863025    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:25.876912 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.876894    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-103.ec2.internal\" not found" node="ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.880798 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.880781    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-103.ec2.internal\" not found" node="ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.960677 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.960653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b05ec092a95b70daec77eec1e81aa44-config\") pod \"kube-apiserver-proxy-ip-10-0-140-103.ec2.internal\" (UID: \"8b05ec092a95b70daec77eec1e81aa44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.960783 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.960680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/156f995d1a4076879b985132336f763d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"156f995d1a4076879b985132336f763d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.960783 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.960739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/156f995d1a4076879b985132336f763d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"156f995d1a4076879b985132336f763d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.960783 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.960757    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b05ec092a95b70daec77eec1e81aa44-config\") pod \"kube-apiserver-proxy-ip-10-0-140-103.ec2.internal\" (UID: \"8b05ec092a95b70daec77eec1e81aa44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.960888 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.960823    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/156f995d1a4076879b985132336f763d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"156f995d1a4076879b985132336f763d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.960888 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:25.960848    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/156f995d1a4076879b985132336f763d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"156f995d1a4076879b985132336f763d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:25.963766 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:25.963751    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:26.064839 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:26.064806    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:26.165678 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:26.165622    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:26.179793 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.179776    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:26.183863 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.183848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:26.266679 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:26.266650    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:26.367128 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:26.367096    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:26.467414 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:26.467367    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:26.567684 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:26.567641    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:26.587261 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.587238    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 04 20:40:26.587463 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.587443    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 20:40:26.667751 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:26.667719    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:26.725420 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.725316    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 20:35:26 +0000 UTC" deadline="2028-03-16 19:18:11.629549552 +0000 UTC"
Aug 04 20:40:26.725420 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.725375    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14158h37m44.904179774s"
Aug 04 20:40:26.759559 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.759537    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 20:40:26.767453 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.767431    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 20:40:26.767873 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:26.767843    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:26.768552 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.768536    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 20:40:26.768637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.768568    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 20:40:26.768637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.768580    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID"
Aug 04 20:40:26.791116 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.791084    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 20:40:26.858495 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:26.858460    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156f995d1a4076879b985132336f763d.slice/crio-72af7ff155d8b16d43a99be2ebc5745db5233c4630c40b0f5cb46ab92842fb55 WatchSource:0}: Error finding container 72af7ff155d8b16d43a99be2ebc5745db5233c4630c40b0f5cb46ab92842fb55: Status 404 returned error can't find the container with id 72af7ff155d8b16d43a99be2ebc5745db5233c4630c40b0f5cb46ab92842fb55
Aug 04 20:40:26.866013 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:26.865995    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 20:40:26.868863 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:26.868845    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:26.873002 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:26.872979    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b05ec092a95b70daec77eec1e81aa44.slice/crio-634a1f68579f4d756cc52ffa62e1ab815d472509896501206b1a74ca6d2cafdb WatchSource:0}: Error finding container 634a1f68579f4d756cc52ffa62e1ab815d472509896501206b1a74ca6d2cafdb: Status 404 returned error can't find the container with id 634a1f68579f4d756cc52ffa62e1ab815d472509896501206b1a74ca6d2cafdb
Aug 04 20:40:26.970083 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:26.970027    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:27.070270 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.070185    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found"
Aug 04 20:40:27.117010 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.116981    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 20:40:27.159419 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.159381    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:27.170660 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.170636    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 20:40:27.171663 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.171645    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal"
Aug 04 20:40:27.190662 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.190640    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 20:40:27.635183 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.635143    2565 apiserver.go:52] "Watching apiserver"
Aug 04 20:40:27.643440 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.643416    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 04 20:40:27.643907 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.643878    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-s7nm2","openshift-image-registry/node-ca-xltcv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal","openshift-multus/multus-4kb2h","openshift-dns/node-resolver-qn2gs","openshift-multus/multus-additional-cni-plugins-pjkt7","openshift-multus/network-metrics-daemon-744vd","openshift-network-diagnostics/network-check-target-v7t6q","openshift-network-operator/iptables-alerter-j5cn7","openshift-ovn-kubernetes/ovnkube-node-gxt5x","kube-system/global-pull-secret-syncer-jq4km","kube-system/konnectivity-agent-qwk98","kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"]
Aug 04 20:40:27.646679 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.646659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:27.646778 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.646748    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:27.648998 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.648974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:27.649099 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.649050    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:27.651459 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.651432    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.653864 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.653840    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.654308 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.654287    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 04 20:40:27.654465 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.654445    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hvqnw\""
Aug 04 20:40:27.654606 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.654587    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 04 20:40:27.654689 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.654671    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 04 20:40:27.656308 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.656286    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qn2gs"
Aug 04 20:40:27.656518 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.656297    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 04 20:40:27.656730 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.656711    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6dnst\""
Aug 04 20:40:27.656835 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.656739    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 04 20:40:27.656891 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.656870    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 04 20:40:27.656985 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.656961    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 04 20:40:27.659704 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.658519    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 04 20:40:27.659704 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.658588    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hwk4g\""
Aug 04 20:40:27.659704 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.658851    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 04 20:40:27.659704 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.659031    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 04 20:40:27.660608 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.660588    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.663427 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.663012    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j5cn7"
Aug 04 20:40:27.663427 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.663051    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 04 20:40:27.663427 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.663085    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xv4lj\""
Aug 04 20:40:27.663427 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.663296    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 04 20:40:27.665507 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.665476    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 04 20:40:27.665606 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.665552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:27.665672 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.665614    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:27.665852 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.665829    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 04 20:40:27.665921 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.665829    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 04 20:40:27.666111 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.666095    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9zz4j\""
Aug 04 20:40:27.667962 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.667939    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xltcv"
Aug 04 20:40:27.671630 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.671576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q86w7\" (UniqueName: \"kubernetes.io/projected/47415b07-ea91-4501-8903-26ff51471613-kube-api-access-q86w7\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.672100 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672073    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:27.672209 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5302fdd4-a446-4010-8115-a4ac2915fcc6-tmp-dir\") pod \"node-resolver-qn2gs\" (UID: \"5302fdd4-a446-4010-8115-a4ac2915fcc6\") " pod="openshift-dns/node-resolver-qn2gs"
Aug 04 20:40:27.672209 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-os-release\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.672209 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672172    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-kubelet-config\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:27.672209 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.672491 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:27.672491 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672240    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 04 20:40:27.672491 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672252    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l8xhf\""
Aug 04 20:40:27.672491 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672384    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 04 20:40:27.672491 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672244    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-hostroot\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.672491 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-cnibin\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.672491 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-dbus\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:27.672829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-socket-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.672829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-device-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.672829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47415b07-ea91-4501-8903-26ff51471613-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.672829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db2bc\" (UniqueName: \"kubernetes.io/projected/371d1e66-d185-408c-96e4-e955fb45f77d-kube-api-access-db2bc\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:27.672829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-sys-fs\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.672829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvngs\" (UniqueName: \"kubernetes.io/projected/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-kube-api-access-cvngs\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.672829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-cni-dir\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.672829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-socket-dir-parent\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.672829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-daemon-config\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.672829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-system-cni-dir\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.672829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672825    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47415b07-ea91-4501-8903-26ff51471613-cni-binary-copy\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-run-multus-certs\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrlq6\" (UniqueName: \"kubernetes.io/projected/f07b810a-cc3a-40b5-affa-6aa75dbd263a-kube-api-access-rrlq6\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672980    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5302fdd4-a446-4010-8115-a4ac2915fcc6-hosts-file\") pod \"node-resolver-qn2gs\" (UID: \"5302fdd4-a446-4010-8115-a4ac2915fcc6\") " pod="openshift-dns/node-resolver-qn2gs"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.672999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-registration-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673015    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-system-cni-dir\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f07b810a-cc3a-40b5-affa-6aa75dbd263a-cni-binary-copy\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-run-k8s-cni-cncf-io\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-run-netns\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-etc-kubernetes\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-conf-dir\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-etc-selinux\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673262    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-cnibin\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673285    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-var-lib-cni-multus\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673309    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-var-lib-kubelet\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.673323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673333    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zklvr\" (UniqueName: \"kubernetes.io/projected/5302fdd4-a446-4010-8115-a4ac2915fcc6-kube-api-access-zklvr\") pod \"node-resolver-qn2gs\" (UID: \"5302fdd4-a446-4010-8115-a4ac2915fcc6\") " pod="openshift-dns/node-resolver-qn2gs"
Aug 04 20:40:27.674244 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673376    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.674244 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-os-release\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.674244 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-var-lib-cni-bin\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.674244 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47415b07-ea91-4501-8903-26ff51471613-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.674244 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.673580    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.674651 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.674610    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 20:40:27.675381 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.675362    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 04 20:40:27.675628 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.675610    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 04 20:40:27.675725 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.675626    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 04 20:40:27.675859 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.675841    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 04 20:40:27.676044 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.676009    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:27.677306 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.677284    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 04 20:40:27.677380 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.677349    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 04 20:40:27.677441 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.677385    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 04 20:40:27.677539 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.677522    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 04 20:40:27.677611 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.677284    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5dbpd\""
Aug 04 20:40:27.677649 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.677630    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4fs62\""
Aug 04 20:40:27.678497 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.678471    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 04 20:40:27.678686 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.678670    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 04 20:40:27.678942 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.678924    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-khxfb\""
Aug 04 20:40:27.701716 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.701698    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-54sjn"
Aug 04 20:40:27.714753 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.714736    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-54sjn"
Aug 04 20:40:27.759365 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.759308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" event={"ID":"8b05ec092a95b70daec77eec1e81aa44","Type":"ContainerStarted","Data":"634a1f68579f4d756cc52ffa62e1ab815d472509896501206b1a74ca6d2cafdb"}
Aug 04 20:40:27.759568 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.759550    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 04 20:40:27.761570 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.761542    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" event={"ID":"156f995d1a4076879b985132336f763d","Type":"ContainerStarted","Data":"72af7ff155d8b16d43a99be2ebc5745db5233c4630c40b0f5cb46ab92842fb55"}
Aug 04 20:40:27.774516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.774483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-hostroot\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.774640 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.774543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-cnibin\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.774640 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.774592    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-systemd-units\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.774768 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.774637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-ovn-node-metrics-cert\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.774768 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.774677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-run\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.774768 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.774685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-cnibin\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.774768 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.774735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-device-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.774969 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.774780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-run-systemd\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.774969 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.774893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-sys-fs\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.775136 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-sys-fs\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.775195 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-cni-dir\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.775241 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-daemon-config\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.775291 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-system-cni-dir\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.775372 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-kubernetes\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.775372 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-host\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.775372 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775341    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-hostroot\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.775519 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-system-cni-dir\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.775519 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-device-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.775614 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775590    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d882eee-e214-4ea9-b271-7d1ac8f3d390-agent-certs\") pod \"konnectivity-agent-qwk98\" (UID: \"4d882eee-e214-4ea9-b271-7d1ac8f3d390\") " pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:27.775662 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47415b07-ea91-4501-8903-26ff51471613-cni-binary-copy\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.775700 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/430b112c-c5d1-4736-916b-6577cdc6464a-host\") pod \"node-ca-xltcv\" (UID: \"430b112c-c5d1-4736-916b-6577cdc6464a\") " pod="openshift-image-registry/node-ca-xltcv"
Aug 04 20:40:27.775737 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-cni-dir\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.775771 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775750    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-log-socket\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.775882 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.775924 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5cdn\" (UniqueName: \"kubernetes.io/projected/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-kube-api-access-t5cdn\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.775968 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-run-multus-certs\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.776001 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.775991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-registration-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.776035 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f07b810a-cc3a-40b5-affa-6aa75dbd263a-cni-binary-copy\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.776076 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-run-k8s-cni-cncf-io\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.776156 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-conf-dir\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.776274 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zklvr\" (UniqueName: \"kubernetes.io/projected/5302fdd4-a446-4010-8115-a4ac2915fcc6-kube-api-access-zklvr\") pod \"node-resolver-qn2gs\" (UID: \"5302fdd4-a446-4010-8115-a4ac2915fcc6\") " pod="openshift-dns/node-resolver-qn2gs"
Aug 04 20:40:27.776339 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776290    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcs6\" (UniqueName: \"kubernetes.io/projected/430b112c-c5d1-4736-916b-6577cdc6464a-kube-api-access-xpcs6\") pod \"node-ca-xltcv\" (UID: \"430b112c-c5d1-4736-916b-6577cdc6464a\") " pod="openshift-image-registry/node-ca-xltcv"
Aug 04 20:40:27.776339 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47415b07-ea91-4501-8903-26ff51471613-cni-binary-copy\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.776339 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae150fa6-5089-41bd-93fa-21442100f54c-etc-tuned\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.776531 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-etc-kubernetes\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.776531 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776404    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-registration-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.776531 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-etc-selinux\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.776531 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-run-k8s-cni-cncf-io\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.776531 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-daemon-config\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.776779 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776538    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-run-multus-certs\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.776779 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-cnibin\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.776919 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776899    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-var-lib-cni-multus\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.776982 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776934    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q86w7\" (UniqueName: \"kubernetes.io/projected/47415b07-ea91-4501-8903-26ff51471613-kube-api-access-q86w7\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.776982 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776970    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-sysctl-conf\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.776982 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.776972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-etc-selinux\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.777137 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f07b810a-cc3a-40b5-affa-6aa75dbd263a-cni-binary-copy\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.777137 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777042    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae150fa6-5089-41bd-93fa-21442100f54c-tmp\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.777137 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-conf-dir\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.777137 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gf9r\" (UniqueName: \"kubernetes.io/projected/ae150fa6-5089-41bd-93fa-21442100f54c-kube-api-access-8gf9r\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.777137 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-var-lib-cni-multus\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.777350 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-etc-kubernetes\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.777350 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-var-lib-cni-bin\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.777350 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-sysconfig\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.777350 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-cnibin\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.777350 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-kubelet\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.777613 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-var-lib-cni-bin\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.777613 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.777613 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5302fdd4-a446-4010-8115-a4ac2915fcc6-tmp-dir\") pod \"node-resolver-qn2gs\" (UID: \"5302fdd4-a446-4010-8115-a4ac2915fcc6\") " pod="openshift-dns/node-resolver-qn2gs"
Aug 04 20:40:27.777613 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-os-release\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.777613 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:27.777848 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/430b112c-c5d1-4736-916b-6577cdc6464a-serviceca\") pod \"node-ca-xltcv\" (UID: \"430b112c-c5d1-4736-916b-6577cdc6464a\") " pod="openshift-image-registry/node-ca-xltcv"
Aug 04 20:40:27.777848 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-slash\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.777848 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-var-lib-openvswitch\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.777848 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777791    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-env-overrides\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.777848 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777825    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-ovnkube-script-lib\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.778085 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-dbus\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:27.778085 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/57b8e90e-1ab9-429c-8732-9afb20f0c3e5-iptables-alerter-script\") pod \"iptables-alerter-j5cn7\" (UID: \"57b8e90e-1ab9-429c-8732-9afb20f0c3e5\") " pod="openshift-network-operator/iptables-alerter-j5cn7"
Aug 04 20:40:27.778085 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55hh\" (UniqueName: \"kubernetes.io/projected/57b8e90e-1ab9-429c-8732-9afb20f0c3e5-kube-api-access-k55hh\") pod \"iptables-alerter-j5cn7\" (UID: \"57b8e90e-1ab9-429c-8732-9afb20f0c3e5\") " pod="openshift-network-operator/iptables-alerter-j5cn7"
Aug 04 20:40:27.778085 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.777990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-modprobe-d\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.778085 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778015    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-ovnkube-config\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.778085 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-socket-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.778372 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47415b07-ea91-4501-8903-26ff51471613-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.778372 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db2bc\" (UniqueName: \"kubernetes.io/projected/371d1e66-d185-408c-96e4-e955fb45f77d-kube-api-access-db2bc\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:27.778372 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778150    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57b8e90e-1ab9-429c-8732-9afb20f0c3e5-host-slash\") pod \"iptables-alerter-j5cn7\" (UID: \"57b8e90e-1ab9-429c-8732-9afb20f0c3e5\") " pod="openshift-network-operator/iptables-alerter-j5cn7"
Aug 04 20:40:27.778372 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-systemd\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.778372 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-sys\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.778372 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778235    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvngs\" (UniqueName: \"kubernetes.io/projected/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-kube-api-access-cvngs\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.778372 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-socket-dir-parent\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.778731 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-node-log\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.778731 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-cni-bin\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.778731 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-cni-netd\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.778731 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778513    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.778731 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5k9c\" (UniqueName: \"kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c\") pod \"network-check-target-v7t6q\" (UID: \"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6\") " pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:27.778731 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrlq6\" (UniqueName: \"kubernetes.io/projected/f07b810a-cc3a-40b5-affa-6aa75dbd263a-kube-api-access-rrlq6\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.778731 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778613    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5302fdd4-a446-4010-8115-a4ac2915fcc6-hosts-file\") pod \"node-resolver-qn2gs\" (UID: \"5302fdd4-a446-4010-8115-a4ac2915fcc6\") " pod="openshift-dns/node-resolver-qn2gs"
Aug 04 20:40:27.778731 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-system-cni-dir\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.778731 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-run-netns\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.778731 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-kubelet-config\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-var-lib-kubelet\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-run-openvswitch\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d882eee-e214-4ea9-b271-7d1ac8f3d390-konnectivity-ca\") pod \"konnectivity-agent-qwk98\" (UID: \"4d882eee-e214-4ea9-b271-7d1ac8f3d390\") " pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-var-lib-kubelet\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47415b07-ea91-4501-8903-26ff51471613-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-sysctl-d\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.778994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-lib-modules\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-etc-openvswitch\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-os-release\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.779206 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-run-ovn\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.779838 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-run-netns\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.779838 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779518    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-dbus\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:27.779838 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-multus-socket-dir-parent\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.779838 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779645    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-os-release\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.779838 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.779739    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:27.780099 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779841    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-system-cni-dir\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.780099 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779849    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5302fdd4-a446-4010-8115-a4ac2915fcc6-hosts-file\") pod \"node-resolver-qn2gs\" (UID: \"5302fdd4-a446-4010-8115-a4ac2915fcc6\") " pod="openshift-dns/node-resolver-qn2gs"
Aug 04 20:40:27.780099 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-run-netns\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.780099 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5302fdd4-a446-4010-8115-a4ac2915fcc6-tmp-dir\") pod \"node-resolver-qn2gs\" (UID: \"5302fdd4-a446-4010-8115-a4ac2915fcc6\") " pod="openshift-dns/node-resolver-qn2gs"
Aug 04 20:40:27.780099 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.779995    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-host-var-lib-kubelet\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.780099 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.780029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.780099 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.780089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f07b810a-cc3a-40b5-affa-6aa75dbd263a-os-release\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.780500 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.780164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-kubelet-config\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:27.780500 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.780191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-socket-dir\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.780500 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.780239    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret podName:79262ffd-10ea-4f77-be9b-0e12ac5a1c53 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:28.280214872 +0000 UTC m=+3.032320834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret") pod "global-pull-secret-syncer-jq4km" (UID: "79262ffd-10ea-4f77-be9b-0e12ac5a1c53") : object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:27.780500 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.780292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47415b07-ea91-4501-8903-26ff51471613-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.780500 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.780319    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:27.780774 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.780506    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47415b07-ea91-4501-8903-26ff51471613-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.780774 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.780540    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs podName:371d1e66-d185-408c-96e4-e955fb45f77d nodeName:}" failed. No retries permitted until 2026-08-04 20:40:28.280518222 +0000 UTC m=+3.032624195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs") pod "network-metrics-daemon-744vd" (UID: "371d1e66-d185-408c-96e4-e955fb45f77d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:27.780774 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.780686    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47415b07-ea91-4501-8903-26ff51471613-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.785340 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.785306    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 04 20:40:27.789459 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.789394    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q86w7\" (UniqueName: \"kubernetes.io/projected/47415b07-ea91-4501-8903-26ff51471613-kube-api-access-q86w7\") pod \"multus-additional-cni-plugins-pjkt7\" (UID: \"47415b07-ea91-4501-8903-26ff51471613\") " pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.790172 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.789955    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrlq6\" (UniqueName: \"kubernetes.io/projected/f07b810a-cc3a-40b5-affa-6aa75dbd263a-kube-api-access-rrlq6\") pod \"multus-4kb2h\" (UID: \"f07b810a-cc3a-40b5-affa-6aa75dbd263a\") " pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.790579 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.790557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zklvr\" (UniqueName: \"kubernetes.io/projected/5302fdd4-a446-4010-8115-a4ac2915fcc6-kube-api-access-zklvr\") pod \"node-resolver-qn2gs\" (UID: \"5302fdd4-a446-4010-8115-a4ac2915fcc6\") " pod="openshift-dns/node-resolver-qn2gs"
Aug 04 20:40:27.791157 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.791121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvngs\" (UniqueName: \"kubernetes.io/projected/e5b904f4-7ac2-4fcc-8ce0-504a600b6014-kube-api-access-cvngs\") pod \"aws-ebs-csi-driver-node-jkltn\" (UID: \"e5b904f4-7ac2-4fcc-8ce0-504a600b6014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.792249 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.792226    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db2bc\" (UniqueName: \"kubernetes.io/projected/371d1e66-d185-408c-96e4-e955fb45f77d-kube-api-access-db2bc\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:27.879549 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-slash\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.879549 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-var-lib-openvswitch\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.879769 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-env-overrides\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.879769 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-ovnkube-script-lib\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.879769 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-slash\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.879769 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-var-lib-openvswitch\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.879769 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/57b8e90e-1ab9-429c-8732-9afb20f0c3e5-iptables-alerter-script\") pod \"iptables-alerter-j5cn7\" (UID: \"57b8e90e-1ab9-429c-8732-9afb20f0c3e5\") " pod="openshift-network-operator/iptables-alerter-j5cn7"
Aug 04 20:40:27.879769 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55hh\" (UniqueName: \"kubernetes.io/projected/57b8e90e-1ab9-429c-8732-9afb20f0c3e5-kube-api-access-k55hh\") pod \"iptables-alerter-j5cn7\" (UID: \"57b8e90e-1ab9-429c-8732-9afb20f0c3e5\") " pod="openshift-network-operator/iptables-alerter-j5cn7"
Aug 04 20:40:27.879769 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-modprobe-d\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-ovnkube-config\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57b8e90e-1ab9-429c-8732-9afb20f0c3e5-host-slash\") pod \"iptables-alerter-j5cn7\" (UID: \"57b8e90e-1ab9-429c-8732-9afb20f0c3e5\") " pod="openshift-network-operator/iptables-alerter-j5cn7"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879846    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-systemd\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879862    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-sys\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-node-log\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-cni-bin\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879909    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-cni-netd\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5k9c\" (UniqueName: \"kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c\") pod \"network-check-target-v7t6q\" (UID: \"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6\") " pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-var-lib-kubelet\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.879985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-run-openvswitch\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d882eee-e214-4ea9-b271-7d1ac8f3d390-konnectivity-ca\") pod \"konnectivity-agent-qwk98\" (UID: \"4d882eee-e214-4ea9-b271-7d1ac8f3d390\") " pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-sysctl-d\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.880045 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-lib-modules\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-etc-openvswitch\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880082    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-run-ovn\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-run-netns\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-systemd-units\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-ovn-node-metrics-cert\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-run\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-run-systemd\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-kubernetes\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-host\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d882eee-e214-4ea9-b271-7d1ac8f3d390-agent-certs\") pod \"konnectivity-agent-qwk98\" (UID: \"4d882eee-e214-4ea9-b271-7d1ac8f3d390\") " pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/430b112c-c5d1-4736-916b-6577cdc6464a-host\") pod \"node-ca-xltcv\" (UID: \"430b112c-c5d1-4736-916b-6577cdc6464a\") " pod="openshift-image-registry/node-ca-xltcv"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-log-socket\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-ovnkube-script-lib\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-node-log\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880332    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-cni-bin\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880354    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-systemd\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-lib-modules\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.880637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-log-socket\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-etc-openvswitch\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5cdn\" (UniqueName: \"kubernetes.io/projected/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-kube-api-access-t5cdn\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-run-ovn\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcs6\" (UniqueName: \"kubernetes.io/projected/430b112c-c5d1-4736-916b-6577cdc6464a-kube-api-access-xpcs6\") pod \"node-ca-xltcv\" (UID: \"430b112c-c5d1-4736-916b-6577cdc6464a\") " pod="openshift-image-registry/node-ca-xltcv"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880551    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-run-netns\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae150fa6-5089-41bd-93fa-21442100f54c-etc-tuned\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880593    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-systemd-units\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-sysctl-conf\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880622    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae150fa6-5089-41bd-93fa-21442100f54c-tmp\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gf9r\" (UniqueName: \"kubernetes.io/projected/ae150fa6-5089-41bd-93fa-21442100f54c-kube-api-access-8gf9r\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-sysconfig\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-kubelet\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880721    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-env-overrides\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/430b112c-c5d1-4736-916b-6577cdc6464a-serviceca\") pod \"node-ca-xltcv\" (UID: \"430b112c-c5d1-4736-916b-6577cdc6464a\") " pod="openshift-image-registry/node-ca-xltcv"
Aug 04 20:40:27.881524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880852    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-ovnkube-config\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880859    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-modprobe-d\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880910    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-cni-netd\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-run\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d882eee-e214-4ea9-b271-7d1ac8f3d390-konnectivity-ca\") pod \"konnectivity-agent-qwk98\" (UID: \"4d882eee-e214-4ea9-b271-7d1ac8f3d390\") " pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-kubernetes\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.880980    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/430b112c-c5d1-4736-916b-6577cdc6464a-host\") pod \"node-ca-xltcv\" (UID: \"430b112c-c5d1-4736-916b-6577cdc6464a\") " pod="openshift-image-registry/node-ca-xltcv"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881008    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57b8e90e-1ab9-429c-8732-9afb20f0c3e5-host-slash\") pod \"iptables-alerter-j5cn7\" (UID: \"57b8e90e-1ab9-429c-8732-9afb20f0c3e5\") " pod="openshift-network-operator/iptables-alerter-j5cn7"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-run-openvswitch\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881098    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-sysctl-d\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-sys\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881159    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-host\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881173    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-var-lib-kubelet\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-kubelet\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881207    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/57b8e90e-1ab9-429c-8732-9afb20f0c3e5-iptables-alerter-script\") pod \"iptables-alerter-j5cn7\" (UID: \"57b8e90e-1ab9-429c-8732-9afb20f0c3e5\") " pod="openshift-network-operator/iptables-alerter-j5cn7"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881218    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-run-systemd\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881256    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-sysconfig\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.882352 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/430b112c-c5d1-4736-916b-6577cdc6464a-serviceca\") pod \"node-ca-xltcv\" (UID: \"430b112c-c5d1-4736-916b-6577cdc6464a\") " pod="openshift-image-registry/node-ca-xltcv"
Aug 04 20:40:27.883049 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.883049 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.881354    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae150fa6-5089-41bd-93fa-21442100f54c-etc-sysctl-conf\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.883535 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.883511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-ovn-node-metrics-cert\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.883764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.883744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae150fa6-5089-41bd-93fa-21442100f54c-etc-tuned\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.884086 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.884067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d882eee-e214-4ea9-b271-7d1ac8f3d390-agent-certs\") pod \"konnectivity-agent-qwk98\" (UID: \"4d882eee-e214-4ea9-b271-7d1ac8f3d390\") " pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:27.884161 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.884106    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae150fa6-5089-41bd-93fa-21442100f54c-tmp\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.889875 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.889742    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 20:40:27.889875 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.889766    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 20:40:27.889875 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.889780    2565 projected.go:196] Error preparing data for projected volume kube-api-access-r5k9c for pod openshift-network-diagnostics/network-check-target-v7t6q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:27.889875 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:27.889838    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c podName:6f4e12ae-0168-4a15-8ac8-3128e78ddeb6 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:28.389820552 +0000 UTC m=+3.141926531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r5k9c" (UniqueName: "kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c") pod "network-check-target-v7t6q" (UID: "6f4e12ae-0168-4a15-8ac8-3128e78ddeb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:27.892043 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.892017    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5cdn\" (UniqueName: \"kubernetes.io/projected/cbb0ba66-4823-42e8-b6c0-4a5c05b932b6-kube-api-access-t5cdn\") pod \"ovnkube-node-gxt5x\" (UID: \"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:27.892780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.892691    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gf9r\" (UniqueName: \"kubernetes.io/projected/ae150fa6-5089-41bd-93fa-21442100f54c-kube-api-access-8gf9r\") pod \"tuned-s7nm2\" (UID: \"ae150fa6-5089-41bd-93fa-21442100f54c\") " pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:27.892780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.892741    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55hh\" (UniqueName: \"kubernetes.io/projected/57b8e90e-1ab9-429c-8732-9afb20f0c3e5-kube-api-access-k55hh\") pod \"iptables-alerter-j5cn7\" (UID: \"57b8e90e-1ab9-429c-8732-9afb20f0c3e5\") " pod="openshift-network-operator/iptables-alerter-j5cn7"
Aug 04 20:40:27.893024 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.892999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcs6\" (UniqueName: \"kubernetes.io/projected/430b112c-c5d1-4736-916b-6577cdc6464a-kube-api-access-xpcs6\") pod \"node-ca-xltcv\" (UID: \"430b112c-c5d1-4736-916b-6577cdc6464a\") " pod="openshift-image-registry/node-ca-xltcv"
Aug 04 20:40:27.962935 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.962905    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn"
Aug 04 20:40:27.973647 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.973622    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4kb2h"
Aug 04 20:40:27.980505 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.980483    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qn2gs"
Aug 04 20:40:27.988058 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.988035    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pjkt7"
Aug 04 20:40:27.995986 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:27.995967    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j5cn7"
Aug 04 20:40:28.007786 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:28.007755    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xltcv"
Aug 04 20:40:28.015658 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:28.015633    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:28.023532 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:28.023511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s7nm2"
Aug 04 20:40:28.031566 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:28.031543    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:28.038251 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:28.038225    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 20:40:28.285101 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:28.285011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:28.285266 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:28.285108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:28.285266 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:28.285176    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:28.285266 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:28.285213    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:28.285266 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:28.285251    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret podName:79262ffd-10ea-4f77-be9b-0e12ac5a1c53 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:29.285232357 +0000 UTC m=+4.037338333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret") pod "global-pull-secret-syncer-jq4km" (UID: "79262ffd-10ea-4f77-be9b-0e12ac5a1c53") : object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:28.285481 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:28.285272    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs podName:371d1e66-d185-408c-96e4-e955fb45f77d nodeName:}" failed. No retries permitted until 2026-08-04 20:40:29.285262347 +0000 UTC m=+4.037368304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs") pod "network-metrics-daemon-744vd" (UID: "371d1e66-d185-408c-96e4-e955fb45f77d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:28.487047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:28.487006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5k9c\" (UniqueName: \"kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c\") pod \"network-check-target-v7t6q\" (UID: \"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6\") " pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:28.487218 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:28.487163    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 20:40:28.487218 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:28.487181    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 20:40:28.487218 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:28.487191    2565 projected.go:196] Error preparing data for projected volume kube-api-access-r5k9c for pod openshift-network-diagnostics/network-check-target-v7t6q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:28.487349 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:28.487256    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c podName:6f4e12ae-0168-4a15-8ac8-3128e78ddeb6 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:29.487238463 +0000 UTC m=+4.239344419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5k9c" (UniqueName: "kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c") pod "network-check-target-v7t6q" (UID: "6f4e12ae-0168-4a15-8ac8-3128e78ddeb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:28.718621 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:28.718530    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 20:35:28 +0000 UTC" deadline="2028-04-02 23:24:47.569855935 +0000 UTC"
Aug 04 20:40:28.718621 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:28.718568    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14570h44m18.851291864s"
Aug 04 20:40:29.013109 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:29.013084    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5302fdd4_a446_4010_8115_a4ac2915fcc6.slice/crio-a59fc5e51314fad7c5e93156e10730e49b40f91dad069dedb82a9a5be1b03615 WatchSource:0}: Error finding container a59fc5e51314fad7c5e93156e10730e49b40f91dad069dedb82a9a5be1b03615: Status 404 returned error can't find the container with id a59fc5e51314fad7c5e93156e10730e49b40f91dad069dedb82a9a5be1b03615
Aug 04 20:40:29.013718 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:29.013692    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d882eee_e214_4ea9_b271_7d1ac8f3d390.slice/crio-7c9adf12f67050ca77ae0d41ba6493f136e1fc6169ae32eee7f8f9cc0fa0f017 WatchSource:0}: Error finding container 7c9adf12f67050ca77ae0d41ba6493f136e1fc6169ae32eee7f8f9cc0fa0f017: Status 404 returned error can't find the container with id 7c9adf12f67050ca77ae0d41ba6493f136e1fc6169ae32eee7f8f9cc0fa0f017
Aug 04 20:40:29.014722 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:29.014441    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430b112c_c5d1_4736_916b_6577cdc6464a.slice/crio-0ee5e3a77ca6cd89252b7c065771741de59a53ed875f0fddb29dc9848a371f71 WatchSource:0}: Error finding container 0ee5e3a77ca6cd89252b7c065771741de59a53ed875f0fddb29dc9848a371f71: Status 404 returned error can't find the container with id 0ee5e3a77ca6cd89252b7c065771741de59a53ed875f0fddb29dc9848a371f71
Aug 04 20:40:29.015352 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:29.015319    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07b810a_cc3a_40b5_affa_6aa75dbd263a.slice/crio-eac1960719cf8d5e5b38069d1994c73ce544c51bb8b9bb31a2c5d9a37920a027 WatchSource:0}: Error finding container eac1960719cf8d5e5b38069d1994c73ce544c51bb8b9bb31a2c5d9a37920a027: Status 404 returned error can't find the container with id eac1960719cf8d5e5b38069d1994c73ce544c51bb8b9bb31a2c5d9a37920a027
Aug 04 20:40:29.019442 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:29.018915    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb0ba66_4823_42e8_b6c0_4a5c05b932b6.slice/crio-27a8954b3aa1a489c76d785cbf2e0882cf784858b03a326caf9d37ac63878c65 WatchSource:0}: Error finding container 27a8954b3aa1a489c76d785cbf2e0882cf784858b03a326caf9d37ac63878c65: Status 404 returned error can't find the container with id 27a8954b3aa1a489c76d785cbf2e0882cf784858b03a326caf9d37ac63878c65
Aug 04 20:40:29.019442 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:29.019210    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b904f4_7ac2_4fcc_8ce0_504a600b6014.slice/crio-ac554fd09000eb7aedf9ece26fc2ba6e1a148601cc79fcfa89b15efa229e0c6f WatchSource:0}: Error finding container ac554fd09000eb7aedf9ece26fc2ba6e1a148601cc79fcfa89b15efa229e0c6f: Status 404 returned error can't find the container with id ac554fd09000eb7aedf9ece26fc2ba6e1a148601cc79fcfa89b15efa229e0c6f
Aug 04 20:40:29.022121 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:40:29.020214    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b8e90e_1ab9_429c_8732_9afb20f0c3e5.slice/crio-6dae4794ff00906235bb245039266f28ca1a4ba450f6b9f46136129868ae427e WatchSource:0}: Error finding container 6dae4794ff00906235bb245039266f28ca1a4ba450f6b9f46136129868ae427e: Status 404 returned error can't find the container with id 6dae4794ff00906235bb245039266f28ca1a4ba450f6b9f46136129868ae427e
Aug 04 20:40:29.294444 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.294267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:29.294444 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:29.294437    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:29.294614 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.294484    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:29.294614 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:29.294509    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret podName:79262ffd-10ea-4f77-be9b-0e12ac5a1c53 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:31.294488433 +0000 UTC m=+6.046594391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret") pod "global-pull-secret-syncer-jq4km" (UID: "79262ffd-10ea-4f77-be9b-0e12ac5a1c53") : object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:29.294614 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:29.294551    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:29.294614 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:29.294599    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs podName:371d1e66-d185-408c-96e4-e955fb45f77d nodeName:}" failed. No retries permitted until 2026-08-04 20:40:31.294582227 +0000 UTC m=+6.046688188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs") pod "network-metrics-daemon-744vd" (UID: "371d1e66-d185-408c-96e4-e955fb45f77d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:29.495781 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.495742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5k9c\" (UniqueName: \"kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c\") pod \"network-check-target-v7t6q\" (UID: \"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6\") " pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:29.496013 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:29.495990    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 20:40:29.496072 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:29.496019    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 20:40:29.496072 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:29.496034    2565 projected.go:196] Error preparing data for projected volume kube-api-access-r5k9c for pod openshift-network-diagnostics/network-check-target-v7t6q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:29.496135 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:29.496098    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c podName:6f4e12ae-0168-4a15-8ac8-3128e78ddeb6 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:31.496078988 +0000 UTC m=+6.248184956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5k9c" (UniqueName: "kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c") pod "network-check-target-v7t6q" (UID: "6f4e12ae-0168-4a15-8ac8-3128e78ddeb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:29.719412 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.719319    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 20:35:28 +0000 UTC" deadline="2028-01-28 15:34:19.695503348 +0000 UTC"
Aug 04 20:40:29.719412 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.719386    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13002h53m49.976133928s"
Aug 04 20:40:29.762399 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.761629    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:29.762399 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:29.761770    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:29.762399 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.762248    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:29.762399 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:29.762347    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:29.763155 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.762867    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:29.763155 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:29.762981    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:29.788287 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.788259    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j5cn7" event={"ID":"57b8e90e-1ab9-429c-8732-9afb20f0c3e5","Type":"ContainerStarted","Data":"6dae4794ff00906235bb245039266f28ca1a4ba450f6b9f46136129868ae427e"}
Aug 04 20:40:29.796270 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.796199    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn" event={"ID":"e5b904f4-7ac2-4fcc-8ce0-504a600b6014","Type":"ContainerStarted","Data":"ac554fd09000eb7aedf9ece26fc2ba6e1a148601cc79fcfa89b15efa229e0c6f"}
Aug 04 20:40:29.800635 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.800596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x" event={"ID":"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6","Type":"ContainerStarted","Data":"27a8954b3aa1a489c76d785cbf2e0882cf784858b03a326caf9d37ac63878c65"}
Aug 04 20:40:29.816543 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.816467    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4kb2h" event={"ID":"f07b810a-cc3a-40b5-affa-6aa75dbd263a","Type":"ContainerStarted","Data":"eac1960719cf8d5e5b38069d1994c73ce544c51bb8b9bb31a2c5d9a37920a027"}
Aug 04 20:40:29.825203 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.825138    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qn2gs" event={"ID":"5302fdd4-a446-4010-8115-a4ac2915fcc6","Type":"ContainerStarted","Data":"a59fc5e51314fad7c5e93156e10730e49b40f91dad069dedb82a9a5be1b03615"}
Aug 04 20:40:29.829986 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.829912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s7nm2" event={"ID":"ae150fa6-5089-41bd-93fa-21442100f54c","Type":"ContainerStarted","Data":"1bd397872513066471e9746ea2a5ee281e0a582684323c5f07f4f7db7c36d201"}
Aug 04 20:40:29.844785 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.844737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" event={"ID":"8b05ec092a95b70daec77eec1e81aa44","Type":"ContainerStarted","Data":"8174722b2481f503af0e67b09781664074ddb9c3613f2c482e773b13049edf66"}
Aug 04 20:40:29.851193 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.851163    2565 generic.go:356] "Generic (PLEG): container finished" podID="156f995d1a4076879b985132336f763d" containerID="587e2b30576b05d4e8814e4c214d1a567e5daf7c2fb120cc716247ab7e0be81e" exitCode=0
Aug 04 20:40:29.851307 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.851245    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" event={"ID":"156f995d1a4076879b985132336f763d","Type":"ContainerDied","Data":"587e2b30576b05d4e8814e4c214d1a567e5daf7c2fb120cc716247ab7e0be81e"}
Aug 04 20:40:29.863021 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.862291    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" podStartSLOduration=2.862271652 podStartE2EDuration="2.862271652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:40:29.861295221 +0000 UTC m=+4.613401204" watchObservedRunningTime="2026-08-04 20:40:29.862271652 +0000 UTC m=+4.614377624"
Aug 04 20:40:29.866229 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.866163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjkt7" event={"ID":"47415b07-ea91-4501-8903-26ff51471613","Type":"ContainerStarted","Data":"57d0d55967b68d98c43d5bfef102201ae91fa7a63033e9ebd9ec42ebe00b2264"}
Aug 04 20:40:29.874809 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.874768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwk98" event={"ID":"4d882eee-e214-4ea9-b271-7d1ac8f3d390","Type":"ContainerStarted","Data":"7c9adf12f67050ca77ae0d41ba6493f136e1fc6169ae32eee7f8f9cc0fa0f017"}
Aug 04 20:40:29.891911 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:29.891885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xltcv" event={"ID":"430b112c-c5d1-4736-916b-6577cdc6464a","Type":"ContainerStarted","Data":"0ee5e3a77ca6cd89252b7c065771741de59a53ed875f0fddb29dc9848a371f71"}
Aug 04 20:40:30.915486 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:30.915389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" event={"ID":"156f995d1a4076879b985132336f763d","Type":"ContainerStarted","Data":"88e594ed8af1af2307cd320413296fa1dbf661bdbe84676ed3bdeb3d3f8bd3b7"}
Aug 04 20:40:30.932196 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:30.931847    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" podStartSLOduration=3.931824401 podStartE2EDuration="3.931824401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:40:30.931219972 +0000 UTC m=+5.683325954" watchObservedRunningTime="2026-08-04 20:40:30.931824401 +0000 UTC m=+5.683930386"
Aug 04 20:40:31.312874 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:31.311998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:31.312874 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:31.312080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:31.312874 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:31.312222    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:31.312874 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:31.312285    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs podName:371d1e66-d185-408c-96e4-e955fb45f77d nodeName:}" failed. No retries permitted until 2026-08-04 20:40:35.312267151 +0000 UTC m=+10.064373111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs") pod "network-metrics-daemon-744vd" (UID: "371d1e66-d185-408c-96e4-e955fb45f77d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:31.312874 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:31.312727    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:31.312874 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:31.312779    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret podName:79262ffd-10ea-4f77-be9b-0e12ac5a1c53 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:35.312763285 +0000 UTC m=+10.064869247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret") pod "global-pull-secret-syncer-jq4km" (UID: "79262ffd-10ea-4f77-be9b-0e12ac5a1c53") : object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:31.514073 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:31.513246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5k9c\" (UniqueName: \"kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c\") pod \"network-check-target-v7t6q\" (UID: \"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6\") " pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:31.514073 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:31.513557    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 20:40:31.514073 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:31.513602    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 20:40:31.514073 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:31.513616    2565 projected.go:196] Error preparing data for projected volume kube-api-access-r5k9c for pod openshift-network-diagnostics/network-check-target-v7t6q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:31.514073 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:31.513681    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c podName:6f4e12ae-0168-4a15-8ac8-3128e78ddeb6 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:35.513660287 +0000 UTC m=+10.265766248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5k9c" (UniqueName: "kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c") pod "network-check-target-v7t6q" (UID: "6f4e12ae-0168-4a15-8ac8-3128e78ddeb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:31.758337 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:31.757465    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:31.758337 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:31.757608    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:31.758337 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:31.757971    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:31.758337 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:31.758062    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:31.758337 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:31.758113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:31.758337 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:31.758184    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:33.756668 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:33.756452    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:33.756668 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:33.756470    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:33.756668 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:33.756551    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:33.756668 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:33.756623    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:33.756668 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:33.756656    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:33.757277 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:33.756749    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:35.344447 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:35.344407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:35.344904 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:35.344474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:35.344904 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:35.344603    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:35.344904 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:35.344657    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret podName:79262ffd-10ea-4f77-be9b-0e12ac5a1c53 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:43.344641879 +0000 UTC m=+18.096747838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret") pod "global-pull-secret-syncer-jq4km" (UID: "79262ffd-10ea-4f77-be9b-0e12ac5a1c53") : object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:35.345077 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:35.344988    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:35.345077 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:35.345031    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs podName:371d1e66-d185-408c-96e4-e955fb45f77d nodeName:}" failed. No retries permitted until 2026-08-04 20:40:43.345017093 +0000 UTC m=+18.097123049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs") pod "network-metrics-daemon-744vd" (UID: "371d1e66-d185-408c-96e4-e955fb45f77d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:35.546174 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:35.546137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5k9c\" (UniqueName: \"kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c\") pod \"network-check-target-v7t6q\" (UID: \"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6\") " pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:35.546363 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:35.546305    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 20:40:35.546363 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:35.546324    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 20:40:35.546363 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:35.546336    2565 projected.go:196] Error preparing data for projected volume kube-api-access-r5k9c for pod openshift-network-diagnostics/network-check-target-v7t6q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:35.546536 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:35.546409    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c podName:6f4e12ae-0168-4a15-8ac8-3128e78ddeb6 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:43.546390893 +0000 UTC m=+18.298496863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5k9c" (UniqueName: "kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c") pod "network-check-target-v7t6q" (UID: "6f4e12ae-0168-4a15-8ac8-3128e78ddeb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:35.757252 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:35.757176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:35.757416 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:35.757306    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:35.757680 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:35.757656    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:35.757778 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:35.757757    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:35.757840 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:35.757808    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:35.757898 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:35.757873    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:37.755129 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:37.755046    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:37.755609 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:37.755052    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:37.755609 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:37.755179    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:37.755609 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:37.755214    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:37.755609 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:37.755326    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:37.755609 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:37.755421    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:39.755103 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:39.755068    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:39.755612 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:39.755077    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:39.755612 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:39.755191    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:39.755612 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:39.755077    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:39.755612 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:39.755282    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:39.755612 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:39.755324    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:41.755260 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:41.755222    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:41.755790 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:41.755232    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:41.755790 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:41.755387    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:41.755790 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:41.755446    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:41.755790 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:41.755232    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:41.755790 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:41.755531    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:43.411441 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:43.411403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:43.411953 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:43.411474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:43.411953 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:43.411562    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:43.411953 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:43.411571    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:43.411953 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:43.411613    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret podName:79262ffd-10ea-4f77-be9b-0e12ac5a1c53 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:59.411599636 +0000 UTC m=+34.163705595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret") pod "global-pull-secret-syncer-jq4km" (UID: "79262ffd-10ea-4f77-be9b-0e12ac5a1c53") : object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:43.411953 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:43.411629    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs podName:371d1e66-d185-408c-96e4-e955fb45f77d nodeName:}" failed. No retries permitted until 2026-08-04 20:40:59.411622221 +0000 UTC m=+34.163728180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs") pod "network-metrics-daemon-744vd" (UID: "371d1e66-d185-408c-96e4-e955fb45f77d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:43.612522 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:43.612490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5k9c\" (UniqueName: \"kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c\") pod \"network-check-target-v7t6q\" (UID: \"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6\") " pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:43.612716 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:43.612689    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 20:40:43.612716 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:43.612714    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 20:40:43.612808 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:43.612728    2565 projected.go:196] Error preparing data for projected volume kube-api-access-r5k9c for pod openshift-network-diagnostics/network-check-target-v7t6q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:43.612808 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:43.612792    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c podName:6f4e12ae-0168-4a15-8ac8-3128e78ddeb6 nodeName:}" failed. No retries permitted until 2026-08-04 20:40:59.61277304 +0000 UTC m=+34.364879014 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5k9c" (UniqueName: "kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c") pod "network-check-target-v7t6q" (UID: "6f4e12ae-0168-4a15-8ac8-3128e78ddeb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:43.755052 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:43.755032    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:43.755141 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:43.755038    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:43.755184 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:43.755138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:43.755231 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:43.755122    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:43.755292 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:43.755235    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:43.755354 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:43.755316    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:45.755236 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:45.755205    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:45.755826 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:45.755277    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:45.755826 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:45.755784    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:45.756013 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:45.755875    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:45.756013 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:45.755308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:45.756013 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:45.755980    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:47.754504 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:47.754475    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:47.755039 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:47.754480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:47.755039 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:47.754610    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:47.755039 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:47.754480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:47.755039 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:47.754691    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:47.755039 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:47.754764    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:48.960378 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.960176    2565 generic.go:356] "Generic (PLEG): container finished" podID="47415b07-ea91-4501-8903-26ff51471613" containerID="bb1cc8406217418c132f47f91456108dd500b8857c9f055d319962cddc9dfa9d" exitCode=0
Aug 04 20:40:48.961287 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.960254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjkt7" event={"ID":"47415b07-ea91-4501-8903-26ff51471613","Type":"ContainerDied","Data":"bb1cc8406217418c132f47f91456108dd500b8857c9f055d319962cddc9dfa9d"}
Aug 04 20:40:48.962205 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.961996    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwk98" event={"ID":"4d882eee-e214-4ea9-b271-7d1ac8f3d390","Type":"ContainerStarted","Data":"bbfce623ace7be516dc2fa753482b77a170ad7c8029b2493ef37c79539722b8a"}
Aug 04 20:40:48.963596 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.963550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xltcv" event={"ID":"430b112c-c5d1-4736-916b-6577cdc6464a","Type":"ContainerStarted","Data":"87baed8f296dfdf31da32371d1f1dc194be34638721189add5e6180fafd2701f"}
Aug 04 20:40:48.965098 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.965070    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn" event={"ID":"e5b904f4-7ac2-4fcc-8ce0-504a600b6014","Type":"ContainerStarted","Data":"e99c720132e1a2d9d7a8fe5b27b1a6015f2d575e74efbdb7c95185e8f0cee5c4"}
Aug 04 20:40:48.968037 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.968009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x" event={"ID":"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6","Type":"ContainerStarted","Data":"ddee370f56b223bf518deeb909b0fb7299cc98ca5ad94d21c431fcdbf7402b8e"}
Aug 04 20:40:48.968140 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.968090    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x" event={"ID":"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6","Type":"ContainerStarted","Data":"1638e343533f8eae47031d8c73be55139dac3d135c0a7995cb66510ba803b8e6"}
Aug 04 20:40:48.968207 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.968142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x" event={"ID":"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6","Type":"ContainerStarted","Data":"6af0168b5a7524c6ff6408a3be71fdd1f1d0ae98812c4db194b8fd58b4a20b50"}
Aug 04 20:40:48.968207 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.968191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x" event={"ID":"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6","Type":"ContainerStarted","Data":"186a46b52ecbcd43a3e290a6421c70e2e95da669840b1178a9ca2e61d529d3ee"}
Aug 04 20:40:48.968315 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.968233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x" event={"ID":"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6","Type":"ContainerStarted","Data":"6e6cb3db66743552f7e8153fd25c1cddfb6b1bb8a9a5ae63a44469552531cf6b"}
Aug 04 20:40:48.968315 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.968274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x" event={"ID":"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6","Type":"ContainerStarted","Data":"eed1bd0ad13678de9ba824fa554d5ec35a58b618b2ff02bb359c320632e8253d"}
Aug 04 20:40:48.969382 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.969357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4kb2h" event={"ID":"f07b810a-cc3a-40b5-affa-6aa75dbd263a","Type":"ContainerStarted","Data":"2162c67d8e121ef4dddf174bae1be904565ca685d8cdd0852ed2b712137c9a17"}
Aug 04 20:40:48.970783 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.970754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qn2gs" event={"ID":"5302fdd4-a446-4010-8115-a4ac2915fcc6","Type":"ContainerStarted","Data":"022fbece5e97dc4e4dbf2e574f38604b0587190f767d029209508353f90c0b51"}
Aug 04 20:40:48.972106 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:48.972082    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s7nm2" event={"ID":"ae150fa6-5089-41bd-93fa-21442100f54c","Type":"ContainerStarted","Data":"061962c60277002ebac941c4dd53eb87a2494c2962be55dac7680298dedbb8f2"}
Aug 04 20:40:49.014265 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.014218    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4kb2h" podStartSLOduration=4.127626606 podStartE2EDuration="23.014204286s" totalImagesPullingTime="18.88657768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:40:49.01415934 +0000 UTC m=+23.766265316" watchObservedRunningTime="2026-08-04 20:40:49.014204286 +0000 UTC m=+23.766310262"
Aug 04 20:40:49.027843 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.027809    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xltcv" podStartSLOduration=4.204723988 podStartE2EDuration="23.027795931s" totalImagesPullingTime="18.823071943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:40:49.02773363 +0000 UTC m=+23.779839606" watchObservedRunningTime="2026-08-04 20:40:49.027795931 +0000 UTC m=+23.779901908"
Aug 04 20:40:49.068881 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.068837    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s7nm2" podStartSLOduration=4.23762296 podStartE2EDuration="23.068824966s" totalImagesPullingTime="18.831202006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:40:49.048027439 +0000 UTC m=+23.800133410" watchObservedRunningTime="2026-08-04 20:40:49.068824966 +0000 UTC m=+23.820930942"
Aug 04 20:40:49.069137 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.069110    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qn2gs" podStartSLOduration=4.247974426 podStartE2EDuration="23.069101245s" totalImagesPullingTime="18.821126819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:40:49.06866355 +0000 UTC m=+23.820769525" watchObservedRunningTime="2026-08-04 20:40:49.069101245 +0000 UTC m=+23.821207221"
Aug 04 20:40:49.432102 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.432074    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 04 20:40:49.715008 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.714940    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:49.715508 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.715491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:49.733470 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.733431    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qwk98" podStartSLOduration=8.998194112 podStartE2EDuration="23.733415919s" totalImagesPullingTime="14.735221807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:40:49.096091393 +0000 UTC m=+23.848197369" watchObservedRunningTime="2026-08-04 20:40:49.733415919 +0000 UTC m=+24.485521895"
Aug 04 20:40:49.738855 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.738785    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T20:40:49.432096353Z","UUID":"7149374e-ecd7-4525-b2b5-506484226aa3","Handler":null,"Name":"","Endpoint":""}
Aug 04 20:40:49.740577 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.740510    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 04 20:40:49.740577 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.740576    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 04 20:40:49.754468 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.754450    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:49.754468 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.754461    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:49.754604 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:49.754533    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:49.754638 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:49.754606    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:49.754681 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.754667    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:49.754764 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:49.754748    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:49.976662 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.976584    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j5cn7" event={"ID":"57b8e90e-1ab9-429c-8732-9afb20f0c3e5","Type":"ContainerStarted","Data":"5216d3635bdf9cc78d2c31eb2c4a1258d716ee3e07e3710500acaa4fdfb91243"}
Aug 04 20:40:49.978669 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.978569    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn" event={"ID":"e5b904f4-7ac2-4fcc-8ce0-504a600b6014","Type":"ContainerStarted","Data":"dc03fe797a98e7ac910cf49f232fb001958b40eadcc6fd3ed7b90e46f6146920"}
Aug 04 20:40:49.993536 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:49.993479    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j5cn7" podStartSLOduration=5.17882318 podStartE2EDuration="23.993465225s" totalImagesPullingTime="18.814642045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:40:49.993116542 +0000 UTC m=+24.745222520" watchObservedRunningTime="2026-08-04 20:40:49.993465225 +0000 UTC m=+24.745571202"
Aug 04 20:40:50.983839 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:50.983749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn" event={"ID":"e5b904f4-7ac2-4fcc-8ce0-504a600b6014","Type":"ContainerStarted","Data":"a175bc0d6208b00aa79a5be19e2ed1720ce705668963ee5e6d3720a78675fba2"}
Aug 04 20:40:50.987359 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:50.987311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x" event={"ID":"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6","Type":"ContainerStarted","Data":"e5e2726f8ff2e73599231a3e9f6c6b40ce6adeb5e7f2b794f655f6ff4249870e"}
Aug 04 20:40:50.987480 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:50.987362    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Aug 04 20:40:51.004964 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:51.004912    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkltn" podStartSLOduration=3.7168963120000003 podStartE2EDuration="25.004894443s" totalImagesPullingTime="21.287998131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:40:51.004251654 +0000 UTC m=+25.756357632" watchObservedRunningTime="2026-08-04 20:40:51.004894443 +0000 UTC m=+25.757000421"
Aug 04 20:40:51.754443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:51.754409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:51.754443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:51.754426    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:51.754667 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:51.754421    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:51.754667 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:51.754535    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:51.754774 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:51.754668    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:51.754774 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:51.754754    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:53.754733 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:53.754581    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:53.755429 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:53.754596    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:53.755429 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:53.754808    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:53.755429 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:53.754919    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:53.755429 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:53.754626    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:53.755429 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:53.755015    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:53.995883 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:53.995811    2565 generic.go:356] "Generic (PLEG): container finished" podID="47415b07-ea91-4501-8903-26ff51471613" containerID="8efc446a3656be3b33410535fa9ca96b81177ba58b574803e266f9b6f11dcd1e" exitCode=0
Aug 04 20:40:53.996029 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:53.995894    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjkt7" event={"ID":"47415b07-ea91-4501-8903-26ff51471613","Type":"ContainerDied","Data":"8efc446a3656be3b33410535fa9ca96b81177ba58b574803e266f9b6f11dcd1e"}
Aug 04 20:40:53.999565 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:53.999443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x" event={"ID":"cbb0ba66-4823-42e8-b6c0-4a5c05b932b6","Type":"ContainerStarted","Data":"82743b61d6e85966c126761dd2e929e2d07ef034dd07c0909fdb00d754121d90"}
Aug 04 20:40:53.999755 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:53.999735    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:53.999871 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:53.999803    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:53.999871 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:53.999850    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:54.014532 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:54.014510    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:54.014659 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:54.014602    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:40:54.051999 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:54.051953    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x" podStartSLOduration=9.163770984 podStartE2EDuration="28.051940948s" totalImagesPullingTime="18.888169964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:40:54.050269778 +0000 UTC m=+28.802375755" watchObservedRunningTime="2026-08-04 20:40:54.051940948 +0000 UTC m=+28.804046924"
Aug 04 20:40:55.529875 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:55.529621    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jq4km"]
Aug 04 20:40:55.530253 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:55.529958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:55.530253 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:55.530051    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:55.535255 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:55.535233    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-744vd"]
Aug 04 20:40:55.535398 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:55.535328    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:55.535456 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:55.535440    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:55.537617 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:55.537592    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v7t6q"]
Aug 04 20:40:55.537719 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:55.537663    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:55.537766 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:55.537734    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:56.005393 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:56.005300    2565 generic.go:356] "Generic (PLEG): container finished" podID="47415b07-ea91-4501-8903-26ff51471613" containerID="5c81cca3c627db989446fb21dcf7a8eed551e334b0a906491e45a8af0ca6b37d" exitCode=0
Aug 04 20:40:56.005393 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:56.005377    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjkt7" event={"ID":"47415b07-ea91-4501-8903-26ff51471613","Type":"ContainerDied","Data":"5c81cca3c627db989446fb21dcf7a8eed551e334b0a906491e45a8af0ca6b37d"}
Aug 04 20:40:56.230403 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:56.230375    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:56.230562 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:56.230508    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Aug 04 20:40:56.231424 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:56.231404    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qwk98"
Aug 04 20:40:56.754360 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:56.754321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:56.754745 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:56.754466    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:56.754745 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:56.754510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:56.754745 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:56.754630    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:57.755169 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:57.755143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:57.755565 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:57.755238    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:40:58.011696 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:58.011612    2565 generic.go:356] "Generic (PLEG): container finished" podID="47415b07-ea91-4501-8903-26ff51471613" containerID="fc1061ad89757ae563d9bdd45e0a03e435434897b3bbf7a3404e1cb49e074637" exitCode=0
Aug 04 20:40:58.011696 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:58.011666    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjkt7" event={"ID":"47415b07-ea91-4501-8903-26ff51471613","Type":"ContainerDied","Data":"fc1061ad89757ae563d9bdd45e0a03e435434897b3bbf7a3404e1cb49e074637"}
Aug 04 20:40:58.754174 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:58.754096    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:58.754328 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:58.754100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:58.754328 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:58.754215    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7t6q" podUID="6f4e12ae-0168-4a15-8ac8-3128e78ddeb6"
Aug 04 20:40:58.754328 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:58.754304    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-744vd" podUID="371d1e66-d185-408c-96e4-e955fb45f77d"
Aug 04 20:40:59.431480 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:59.431448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:59.431927 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:59.431511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:40:59.431927 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:59.431689    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:59.431927 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:59.431705    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:59.431927 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:59.431743    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret podName:79262ffd-10ea-4f77-be9b-0e12ac5a1c53 nodeName:}" failed. No retries permitted until 2026-08-04 20:41:31.431724997 +0000 UTC m=+66.183830964 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret") pod "global-pull-secret-syncer-jq4km" (UID: "79262ffd-10ea-4f77-be9b-0e12ac5a1c53") : object "kube-system"/"original-pull-secret" not registered
Aug 04 20:40:59.431927 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:59.431785    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs podName:371d1e66-d185-408c-96e4-e955fb45f77d nodeName:}" failed. No retries permitted until 2026-08-04 20:41:31.431764742 +0000 UTC m=+66.183870704 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs") pod "network-metrics-daemon-744vd" (UID: "371d1e66-d185-408c-96e4-e955fb45f77d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 20:40:59.632560 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:59.632519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5k9c\" (UniqueName: \"kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c\") pod \"network-check-target-v7t6q\" (UID: \"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6\") " pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:40:59.632733 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:59.632674    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 20:40:59.632733 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:59.632698    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 20:40:59.632733 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:59.632709    2565 projected.go:196] Error preparing data for projected volume kube-api-access-r5k9c for pod openshift-network-diagnostics/network-check-target-v7t6q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:59.632878 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:59.632764    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c podName:6f4e12ae-0168-4a15-8ac8-3128e78ddeb6 nodeName:}" failed. No retries permitted until 2026-08-04 20:41:31.632750855 +0000 UTC m=+66.384856829 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5k9c" (UniqueName: "kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c") pod "network-check-target-v7t6q" (UID: "6f4e12ae-0168-4a15-8ac8-3128e78ddeb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 20:40:59.754684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:40:59.754604    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:40:59.754828 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:40:59.754737    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jq4km" podUID="79262ffd-10ea-4f77-be9b-0e12ac5a1c53"
Aug 04 20:41:00.527490 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.527307    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeReady"
Aug 04 20:41:00.527947 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.527622    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 04 20:41:00.564842 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.564799    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fbd494568-8vch9"]
Aug 04 20:41:00.568942 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.568927    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.571579 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.571554    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 04 20:41:00.571701 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.571579    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 04 20:41:00.571701 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.571582    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 04 20:41:00.571701 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.571673    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sm2np\""
Aug 04 20:41:00.577150 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.577132    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 04 20:41:00.578114 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.578091    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zr7n9"]
Aug 04 20:41:00.581938 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.581921    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.582873 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.582847    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fbd494568-8vch9"]
Aug 04 20:41:00.583360 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.583326    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4nrkd"]
Aug 04 20:41:00.584451 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.584425    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 04 20:41:00.584547 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.584492    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kcgl6\""
Aug 04 20:41:00.584617 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.584567    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 04 20:41:00.586969 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.586951    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4nrkd"
Aug 04 20:41:00.589538 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.589520    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 04 20:41:00.589538 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.589537    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q6crk\""
Aug 04 20:41:00.589672 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.589651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 04 20:41:00.589773 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.589760    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 04 20:41:00.593555 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.593536    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zr7n9"]
Aug 04 20:41:00.598386 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.597005    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4nrkd"]
Aug 04 20:41:00.739922 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.739835    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.739922 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.739879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grns2\" (UniqueName: \"kubernetes.io/projected/cbe77e45-7755-4781-93b4-75fe732d1507-kube-api-access-grns2\") pod \"ingress-canary-4nrkd\" (UID: \"cbe77e45-7755-4781-93b4-75fe732d1507\") " pod="openshift-ingress-canary/ingress-canary-4nrkd"
Aug 04 20:41:00.739922 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.739908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8c1ad3c-f203-48b1-9368-866418f92fc5-ca-trust-extracted\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.740173 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.739983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.740173 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.740031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f2150f-4332-4f28-abe4-72de91210bfa-tmp-dir\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.740173 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.740055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8fpm\" (UniqueName: \"kubernetes.io/projected/d6f2150f-4332-4f28-abe4-72de91210bfa-kube-api-access-r8fpm\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.740173 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.740085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-image-registry-private-configuration\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.740173 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.740149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbe77e45-7755-4781-93b4-75fe732d1507-cert\") pod \"ingress-canary-4nrkd\" (UID: \"cbe77e45-7755-4781-93b4-75fe732d1507\") " pod="openshift-ingress-canary/ingress-canary-4nrkd"
Aug 04 20:41:00.740371 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.740204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f2150f-4332-4f28-abe4-72de91210bfa-config-volume\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.740371 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.740224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-certificates\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.740371 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.740247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-trusted-ca\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.740371 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.740273    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-installation-pull-secrets\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.740371 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.740294    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-bound-sa-token\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.740371 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.740316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5fh\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-kube-api-access-px5fh\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.754324 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.754295    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:41:00.754324 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.754265    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:41:00.757360 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.757322    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 20:41:00.757477 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.757362    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 20:41:00.757477 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.757410    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 20:41:00.757636 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.757621    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h8x4x\""
Aug 04 20:41:00.757813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.757791    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nn596\""
Aug 04 20:41:00.841492 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f2150f-4332-4f28-abe4-72de91210bfa-config-volume\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.841620 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841521    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-certificates\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.841620 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-trusted-ca\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.841620 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841589    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-installation-pull-secrets\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.841780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-bound-sa-token\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.841780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841647    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5fh\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-kube-api-access-px5fh\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.841780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.841780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grns2\" (UniqueName: \"kubernetes.io/projected/cbe77e45-7755-4781-93b4-75fe732d1507-kube-api-access-grns2\") pod \"ingress-canary-4nrkd\" (UID: \"cbe77e45-7755-4781-93b4-75fe732d1507\") " pod="openshift-ingress-canary/ingress-canary-4nrkd"
Aug 04 20:41:00.841780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8c1ad3c-f203-48b1-9368-866418f92fc5-ca-trust-extracted\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.841780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.842038 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f2150f-4332-4f28-abe4-72de91210bfa-tmp-dir\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.842038 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8fpm\" (UniqueName: \"kubernetes.io/projected/d6f2150f-4332-4f28-abe4-72de91210bfa-kube-api-access-r8fpm\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.842038 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-image-registry-private-configuration\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.842038 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.841882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbe77e45-7755-4781-93b4-75fe732d1507-cert\") pod \"ingress-canary-4nrkd\" (UID: \"cbe77e45-7755-4781-93b4-75fe732d1507\") " pod="openshift-ingress-canary/ingress-canary-4nrkd"
Aug 04 20:41:00.842200 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:00.842061    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 20:41:00.842200 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:00.842078    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbd494568-8vch9: secret "image-registry-tls" not found
Aug 04 20:41:00.842200 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.842138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f2150f-4332-4f28-abe4-72de91210bfa-config-volume\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.842200 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:00.842154    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls podName:a8c1ad3c-f203-48b1-9368-866418f92fc5 nodeName:}" failed. No retries permitted until 2026-08-04 20:41:01.342134702 +0000 UTC m=+36.094240670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls") pod "image-registry-5fbd494568-8vch9" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5") : secret "image-registry-tls" not found
Aug 04 20:41:00.842200 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.842142    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8c1ad3c-f203-48b1-9368-866418f92fc5-ca-trust-extracted\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.842480 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:00.842403    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 20:41:00.842480 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.842448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f2150f-4332-4f28-abe4-72de91210bfa-tmp-dir\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.842562 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:00.842509    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls podName:d6f2150f-4332-4f28-abe4-72de91210bfa nodeName:}" failed. No retries permitted until 2026-08-04 20:41:01.342489434 +0000 UTC m=+36.094595398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls") pod "dns-default-zr7n9" (UID: "d6f2150f-4332-4f28-abe4-72de91210bfa") : secret "dns-default-metrics-tls" not found
Aug 04 20:41:00.842609 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.842554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-certificates\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.842853 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.842830    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-trusted-ca\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.846194 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.846169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-image-registry-private-configuration\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.846194 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.846183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-installation-pull-secrets\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.846337 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.846189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbe77e45-7755-4781-93b4-75fe732d1507-cert\") pod \"ingress-canary-4nrkd\" (UID: \"cbe77e45-7755-4781-93b4-75fe732d1507\") " pod="openshift-ingress-canary/ingress-canary-4nrkd"
Aug 04 20:41:00.854676 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.854649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-bound-sa-token\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.856224 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.856204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8fpm\" (UniqueName: \"kubernetes.io/projected/d6f2150f-4332-4f28-abe4-72de91210bfa-kube-api-access-r8fpm\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:00.856394 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.856255    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grns2\" (UniqueName: \"kubernetes.io/projected/cbe77e45-7755-4781-93b4-75fe732d1507-kube-api-access-grns2\") pod \"ingress-canary-4nrkd\" (UID: \"cbe77e45-7755-4781-93b4-75fe732d1507\") " pod="openshift-ingress-canary/ingress-canary-4nrkd"
Aug 04 20:41:00.856751 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.856734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5fh\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-kube-api-access-px5fh\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:00.898371 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:00.898330    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4nrkd"
Aug 04 20:41:01.347633 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:01.347593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:01.347633 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:01.347635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:01.347882 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:01.347770    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 20:41:01.347882 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:01.347847    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls podName:d6f2150f-4332-4f28-abe4-72de91210bfa nodeName:}" failed. No retries permitted until 2026-08-04 20:41:02.347829522 +0000 UTC m=+37.099935490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls") pod "dns-default-zr7n9" (UID: "d6f2150f-4332-4f28-abe4-72de91210bfa") : secret "dns-default-metrics-tls" not found
Aug 04 20:41:01.347882 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:01.347770    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 20:41:01.348035 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:01.347890    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbd494568-8vch9: secret "image-registry-tls" not found
Aug 04 20:41:01.348035 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:01.347958    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls podName:a8c1ad3c-f203-48b1-9368-866418f92fc5 nodeName:}" failed. No retries permitted until 2026-08-04 20:41:02.347934688 +0000 UTC m=+37.100040667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls") pod "image-registry-5fbd494568-8vch9" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5") : secret "image-registry-tls" not found
Aug 04 20:41:01.754607 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:01.754574    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:41:01.757556 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:01.757534    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 20:41:02.356618 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:02.356577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:02.356618 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:02.356631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:02.356855 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:02.356725    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 20:41:02.356855 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:02.356742    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 20:41:02.356855 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:02.356764    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbd494568-8vch9: secret "image-registry-tls" not found
Aug 04 20:41:02.356855 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:02.356777    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls podName:d6f2150f-4332-4f28-abe4-72de91210bfa nodeName:}" failed. No retries permitted until 2026-08-04 20:41:04.35676411 +0000 UTC m=+39.108870066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls") pod "dns-default-zr7n9" (UID: "d6f2150f-4332-4f28-abe4-72de91210bfa") : secret "dns-default-metrics-tls" not found
Aug 04 20:41:02.356855 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:02.356828    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls podName:a8c1ad3c-f203-48b1-9368-866418f92fc5 nodeName:}" failed. No retries permitted until 2026-08-04 20:41:04.356811357 +0000 UTC m=+39.108917314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls") pod "image-registry-5fbd494568-8vch9" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5") : secret "image-registry-tls" not found
Aug 04 20:41:02.459287 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:02.459253    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4nrkd"]
Aug 04 20:41:02.464333 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:41:02.464304    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe77e45_7755_4781_93b4_75fe732d1507.slice/crio-9e91d6241d058cfc0f9797fddcd51c38c368fb9c3dbfbbb7bb5544a0bb25b989 WatchSource:0}: Error finding container 9e91d6241d058cfc0f9797fddcd51c38c368fb9c3dbfbbb7bb5544a0bb25b989: Status 404 returned error can't find the container with id 9e91d6241d058cfc0f9797fddcd51c38c368fb9c3dbfbbb7bb5544a0bb25b989
Aug 04 20:41:03.025784 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:03.025594    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4nrkd" event={"ID":"cbe77e45-7755-4781-93b4-75fe732d1507","Type":"ContainerStarted","Data":"9e91d6241d058cfc0f9797fddcd51c38c368fb9c3dbfbbb7bb5544a0bb25b989"}
Aug 04 20:41:03.028225 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:03.028199    2565 generic.go:356] "Generic (PLEG): container finished" podID="47415b07-ea91-4501-8903-26ff51471613" containerID="dab4301bf3703fc8867b17ba5728c88468bc30cb0c6d7ce5107fb929304e8fdd" exitCode=0
Aug 04 20:41:03.028335 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:03.028231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjkt7" event={"ID":"47415b07-ea91-4501-8903-26ff51471613","Type":"ContainerDied","Data":"dab4301bf3703fc8867b17ba5728c88468bc30cb0c6d7ce5107fb929304e8fdd"}
Aug 04 20:41:04.033297 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:04.033262    2565 generic.go:356] "Generic (PLEG): container finished" podID="47415b07-ea91-4501-8903-26ff51471613" containerID="7fc41a7760c05881184cbbe5d6e183ac36c45a090a896927a72fe02a0569e779" exitCode=0
Aug 04 20:41:04.033758 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:04.033319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjkt7" event={"ID":"47415b07-ea91-4501-8903-26ff51471613","Type":"ContainerDied","Data":"7fc41a7760c05881184cbbe5d6e183ac36c45a090a896927a72fe02a0569e779"}
Aug 04 20:41:04.372176 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:04.372102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:04.372309 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:04.372187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:04.372309 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:04.372244    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 20:41:04.372309 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:04.372274    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 20:41:04.372309 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:04.372285    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbd494568-8vch9: secret "image-registry-tls" not found
Aug 04 20:41:04.372450 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:04.372317    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls podName:d6f2150f-4332-4f28-abe4-72de91210bfa nodeName:}" failed. No retries permitted until 2026-08-04 20:41:08.372300979 +0000 UTC m=+43.124406953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls") pod "dns-default-zr7n9" (UID: "d6f2150f-4332-4f28-abe4-72de91210bfa") : secret "dns-default-metrics-tls" not found
Aug 04 20:41:04.372450 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:04.372331    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls podName:a8c1ad3c-f203-48b1-9368-866418f92fc5 nodeName:}" failed. No retries permitted until 2026-08-04 20:41:08.372325908 +0000 UTC m=+43.124431863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls") pod "image-registry-5fbd494568-8vch9" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5") : secret "image-registry-tls" not found
Aug 04 20:41:05.040480 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:05.040442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjkt7" event={"ID":"47415b07-ea91-4501-8903-26ff51471613","Type":"ContainerStarted","Data":"e61cfb1a7d08046c30097a09b99d57d634d9d543bba3edc997e20d3604c9a208"}
Aug 04 20:41:05.066205 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:05.066134    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pjkt7" podStartSLOduration=8.880392493 podStartE2EDuration="39.066116653s" totalImagesPullingTime="30.18572416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:41:05.064246506 +0000 UTC m=+39.816352500" watchObservedRunningTime="2026-08-04 20:41:05.066116653 +0000 UTC m=+39.818222659"
Aug 04 20:41:05.215547 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:05.215515    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:05.311485 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:05.311415    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:06.036401 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:06.036221    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:06.043859 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:06.043827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4nrkd" event={"ID":"cbe77e45-7755-4781-93b4-75fe732d1507","Type":"ContainerStarted","Data":"7dc7b6a768b527dc20d9fd4bc0132d5b4574f91711e2dbcd51f29c9fa484d37d"}
Aug 04 20:41:06.061798 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:06.061729    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4nrkd" podStartSLOduration=3.131396256 podStartE2EDuration="6.061715608s" totalImagesPullingTime="2.930319352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:41:06.061369004 +0000 UTC m=+40.813474986" watchObservedRunningTime="2026-08-04 20:41:06.061715608 +0000 UTC m=+40.813821584"
Aug 04 20:41:07.258180 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:07.258150    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:07.332267 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:07.332234    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:07.396554 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:07.396524    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:07.499017 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:07.498986    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:07.604645 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:07.604572    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:07.697694 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:07.697664    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:08.239998 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:08.239971    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:08.340456 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:08.340429    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:08.397787 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:08.397764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:08.397910 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:08.397832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:08.397910 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:08.397900    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 20:41:08.397991 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:08.397910    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 20:41:08.397991 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:08.397920    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbd494568-8vch9: secret "image-registry-tls" not found
Aug 04 20:41:08.397991 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:08.397956    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls podName:d6f2150f-4332-4f28-abe4-72de91210bfa nodeName:}" failed. No retries permitted until 2026-08-04 20:41:16.397939826 +0000 UTC m=+51.150045784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls") pod "dns-default-zr7n9" (UID: "d6f2150f-4332-4f28-abe4-72de91210bfa") : secret "dns-default-metrics-tls" not found
Aug 04 20:41:08.397991 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:08.397969    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls podName:a8c1ad3c-f203-48b1-9368-866418f92fc5 nodeName:}" failed. No retries permitted until 2026-08-04 20:41:16.397963223 +0000 UTC m=+51.150069178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls") pod "image-registry-5fbd494568-8vch9" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5") : secret "image-registry-tls" not found
Aug 04 20:41:08.638757 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:08.638679    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:08.740251 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:08.740215    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:08.809844 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:08.809815    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:09.435092 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:09.435056    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:09.536067 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:09.536035    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:09.850934 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:09.850907    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:09.950897 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:09.950865    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:15.529156 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:15.529124    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:15.598527 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:15.598499    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:16.209249 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:16.209218    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:16.451328 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:16.451290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:16.451328 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:16.451331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:16.451534 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:16.451433    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 20:41:16.451534 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:16.451441    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 20:41:16.451534 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:16.451458    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbd494568-8vch9: secret "image-registry-tls" not found
Aug 04 20:41:16.451534 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:16.451496    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls podName:d6f2150f-4332-4f28-abe4-72de91210bfa nodeName:}" failed. No retries permitted until 2026-08-04 20:41:32.451481218 +0000 UTC m=+67.203587177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls") pod "dns-default-zr7n9" (UID: "d6f2150f-4332-4f28-abe4-72de91210bfa") : secret "dns-default-metrics-tls" not found
Aug 04 20:41:16.451534 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:16.451508    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls podName:a8c1ad3c-f203-48b1-9368-866418f92fc5 nodeName:}" failed. No retries permitted until 2026-08-04 20:41:32.45150218 +0000 UTC m=+67.203608135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls") pod "image-registry-5fbd494568-8vch9" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5") : secret "image-registry-tls" not found
Aug 04 20:41:17.400232 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:17.400199    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:19.391660 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:19.391611    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:20.557613 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:20.557579    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:26.015707 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:26.015678    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxt5x"
Aug 04 20:41:26.172091 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:26.172050    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:26.236688 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:26.236655    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:28.694852 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:28.694800    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:28.795560 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:28.795523    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log" limit=1024
Aug 04 20:41:31.460649 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.460615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:41:31.461034 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.460673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:41:31.463603 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.463585    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 20:41:31.464413 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.464391    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 20:41:31.471421 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:31.471405    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 04 20:41:31.471518 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:31.471457    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs podName:371d1e66-d185-408c-96e4-e955fb45f77d nodeName:}" failed. No retries permitted until 2026-08-04 20:42:35.471441771 +0000 UTC m=+130.223547726 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs") pod "network-metrics-daemon-744vd" (UID: "371d1e66-d185-408c-96e4-e955fb45f77d") : secret "metrics-daemon-secret" not found
Aug 04 20:41:31.475041 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.475022    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79262ffd-10ea-4f77-be9b-0e12ac5a1c53-original-pull-secret\") pod \"global-pull-secret-syncer-jq4km\" (UID: \"79262ffd-10ea-4f77-be9b-0e12ac5a1c53\") " pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:41:31.662393 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.662337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5k9c\" (UniqueName: \"kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c\") pod \"network-check-target-v7t6q\" (UID: \"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6\") " pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:41:31.665428 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.665405    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 20:41:31.676066 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.676046    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 20:41:31.686571 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.686547    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5k9c\" (UniqueName: \"kubernetes.io/projected/6f4e12ae-0168-4a15-8ac8-3128e78ddeb6-kube-api-access-r5k9c\") pod \"network-check-target-v7t6q\" (UID: \"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6\") " pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:41:31.764770 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.764748    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq4km"
Aug 04 20:41:31.902662 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.902628    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jq4km"]
Aug 04 20:41:31.906235 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:41:31.906207    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79262ffd_10ea_4f77_be9b_0e12ac5a1c53.slice/crio-07bd519bc339dee23801c7554f493c4bed76918b8f9cf4a0eb00920372a9015a WatchSource:0}: Error finding container 07bd519bc339dee23801c7554f493c4bed76918b8f9cf4a0eb00920372a9015a: Status 404 returned error can't find the container with id 07bd519bc339dee23801c7554f493c4bed76918b8f9cf4a0eb00920372a9015a
Aug 04 20:41:31.969248 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.969217    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nn596\""
Aug 04 20:41:31.976662 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:31.976640    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:41:32.100908 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:32.100866    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v7t6q"]
Aug 04 20:41:32.103995 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:41:32.103966    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4e12ae_0168_4a15_8ac8_3128e78ddeb6.slice/crio-5973b36987d755ac1b9eac897753a0bbe538155b08f5bd912c10835123a10be6 WatchSource:0}: Error finding container 5973b36987d755ac1b9eac897753a0bbe538155b08f5bd912c10835123a10be6: Status 404 returned error can't find the container with id 5973b36987d755ac1b9eac897753a0bbe538155b08f5bd912c10835123a10be6
Aug 04 20:41:32.105103 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:32.105078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jq4km" event={"ID":"79262ffd-10ea-4f77-be9b-0e12ac5a1c53","Type":"ContainerStarted","Data":"07bd519bc339dee23801c7554f493c4bed76918b8f9cf4a0eb00920372a9015a"}
Aug 04 20:41:32.466954 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:32.466924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:41:32.467282 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:32.466973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:41:32.467282 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:32.467075    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 20:41:32.467282 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:32.467080    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 20:41:32.467282 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:32.467095    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbd494568-8vch9: secret "image-registry-tls" not found
Aug 04 20:41:32.467282 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:32.467142    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls podName:a8c1ad3c-f203-48b1-9368-866418f92fc5 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:04.467121506 +0000 UTC m=+99.219227475 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls") pod "image-registry-5fbd494568-8vch9" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5") : secret "image-registry-tls" not found
Aug 04 20:41:32.467282 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:41:32.467157    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls podName:d6f2150f-4332-4f28-abe4-72de91210bfa nodeName:}" failed. No retries permitted until 2026-08-04 20:42:04.467149808 +0000 UTC m=+99.219255766 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls") pod "dns-default-zr7n9" (UID: "d6f2150f-4332-4f28-abe4-72de91210bfa") : secret "dns-default-metrics-tls" not found
Aug 04 20:41:33.109462 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:33.109386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v7t6q" event={"ID":"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6","Type":"ContainerStarted","Data":"5973b36987d755ac1b9eac897753a0bbe538155b08f5bd912c10835123a10be6"}
Aug 04 20:41:37.122660 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:37.122621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jq4km" event={"ID":"79262ffd-10ea-4f77-be9b-0e12ac5a1c53","Type":"ContainerStarted","Data":"7c16beec1f9ac04a1765e54f45f76135fc2b6b53a2f11fe36872501eb2707865"}
Aug 04 20:41:37.124061 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:37.124038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v7t6q" event={"ID":"6f4e12ae-0168-4a15-8ac8-3128e78ddeb6","Type":"ContainerStarted","Data":"33e0f0fbc54f3f9760d952667216ef29573cf7615bfc88b5139eb09a9018a496"}
Aug 04 20:41:37.124179 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:37.124165    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:41:37.142623 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:41:37.142546    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jq4km" podStartSLOduration=66.161348179 podStartE2EDuration="1m11.142532599s" totalImagesPullingTime="4.98118442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:41:37.141437356 +0000 UTC m=+71.893543334" watchObservedRunningTime="2026-08-04 20:41:37.142532599 +0000 UTC m=+71.894638575"
Aug 04 20:42:01.899760 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.899700    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v7t6q" podStartSLOduration=91.121178151 podStartE2EDuration="1m35.899681645s" totalImagesPullingTime="4.778503494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:41:37.163691776 +0000 UTC m=+71.915797752" watchObservedRunningTime="2026-08-04 20:42:01.899681645 +0000 UTC m=+96.651787605"
Aug 04 20:42:01.900462 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.900443    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n2rgn"]
Aug 04 20:42:01.904762 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.904740    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:01.907262 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.907233    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 04 20:42:01.907581 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.907560    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 04 20:42:01.908296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.908277    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 04 20:42:01.908419 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.908319    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d7shv\""
Aug 04 20:42:01.908419 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.908319    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 04 20:42:01.925367 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.925326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n2rgn"]
Aug 04 20:42:01.971209 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.971170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:01.971410 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.971232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13da4fd0-9ad4-4721-88fa-7555047c7b69-data-volume\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:01.971410 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.971253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf7sv\" (UniqueName: \"kubernetes.io/projected/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-api-access-hf7sv\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:01.971410 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.971358    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13da4fd0-9ad4-4721-88fa-7555047c7b69-crio-socket\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:01.971518 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:01.971406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:02.072598 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:02.072564    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:02.072706 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:02.072623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13da4fd0-9ad4-4721-88fa-7555047c7b69-data-volume\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:02.072743 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:02.072692    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 20:42:02.072743 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:02.072722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf7sv\" (UniqueName: \"kubernetes.io/projected/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-api-access-hf7sv\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:02.072820 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:02.072753    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13da4fd0-9ad4-4721-88fa-7555047c7b69-crio-socket\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:02.072820 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:02.072776    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm podName:13da4fd0-9ad4-4721-88fa-7555047c7b69 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:02.572755665 +0000 UTC m=+97.324861635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2rgn" (UID: "13da4fd0-9ad4-4721-88fa-7555047c7b69") : configmap "kube-rbac-proxy" not found
Aug 04 20:42:02.072820 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:02.072805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:02.072958 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:02.072815    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13da4fd0-9ad4-4721-88fa-7555047c7b69-crio-socket\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:02.072958 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:02.072886    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13da4fd0-9ad4-4721-88fa-7555047c7b69-data-volume\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:02.072958 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:02.072933    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:02.073154 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:02.072970    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls podName:13da4fd0-9ad4-4721-88fa-7555047c7b69 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:02.572960941 +0000 UTC m=+97.325066900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2rgn" (UID: "13da4fd0-9ad4-4721-88fa-7555047c7b69") : secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:02.083654 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:02.083626    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf7sv\" (UniqueName: \"kubernetes.io/projected/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-api-access-hf7sv\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:02.577694 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:02.577649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:02.577875 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:02.577714    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:02.577875 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:02.577772    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 20:42:02.577875 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:02.577804    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:02.577875 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:02.577858    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm podName:13da4fd0-9ad4-4721-88fa-7555047c7b69 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:03.57784122 +0000 UTC m=+98.329947176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2rgn" (UID: "13da4fd0-9ad4-4721-88fa-7555047c7b69") : configmap "kube-rbac-proxy" not found
Aug 04 20:42:02.577875 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:02.577873    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls podName:13da4fd0-9ad4-4721-88fa-7555047c7b69 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:03.577866619 +0000 UTC m=+98.329972574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2rgn" (UID: "13da4fd0-9ad4-4721-88fa-7555047c7b69") : secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:03.318650 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.318618    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"]
Aug 04 20:42:03.321577 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.321561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:03.325122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.325098    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 04 20:42:03.325122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.325108    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 04 20:42:03.325309 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.325099    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Aug 04 20:42:03.325478 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.325458    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 04 20:42:03.335664 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.335641    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"]
Aug 04 20:42:03.384182 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.384147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a7649cf-3f88-4bab-9a76-b824325a9610-tmp\") pod \"klusterlet-addon-workmgr-7cc75787b9-rn6c4\" (UID: \"2a7649cf-3f88-4bab-9a76-b824325a9610\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:03.384388 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.384211    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2a7649cf-3f88-4bab-9a76-b824325a9610-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cc75787b9-rn6c4\" (UID: \"2a7649cf-3f88-4bab-9a76-b824325a9610\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:03.384388 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.384319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crtj\" (UniqueName: \"kubernetes.io/projected/2a7649cf-3f88-4bab-9a76-b824325a9610-kube-api-access-7crtj\") pod \"klusterlet-addon-workmgr-7cc75787b9-rn6c4\" (UID: \"2a7649cf-3f88-4bab-9a76-b824325a9610\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:03.485400 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.485363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crtj\" (UniqueName: \"kubernetes.io/projected/2a7649cf-3f88-4bab-9a76-b824325a9610-kube-api-access-7crtj\") pod \"klusterlet-addon-workmgr-7cc75787b9-rn6c4\" (UID: \"2a7649cf-3f88-4bab-9a76-b824325a9610\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:03.485532 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.485428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a7649cf-3f88-4bab-9a76-b824325a9610-tmp\") pod \"klusterlet-addon-workmgr-7cc75787b9-rn6c4\" (UID: \"2a7649cf-3f88-4bab-9a76-b824325a9610\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:03.485532 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.485495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2a7649cf-3f88-4bab-9a76-b824325a9610-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cc75787b9-rn6c4\" (UID: \"2a7649cf-3f88-4bab-9a76-b824325a9610\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:03.485822 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.485800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a7649cf-3f88-4bab-9a76-b824325a9610-tmp\") pod \"klusterlet-addon-workmgr-7cc75787b9-rn6c4\" (UID: \"2a7649cf-3f88-4bab-9a76-b824325a9610\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:03.488008 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.487983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2a7649cf-3f88-4bab-9a76-b824325a9610-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cc75787b9-rn6c4\" (UID: \"2a7649cf-3f88-4bab-9a76-b824325a9610\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:03.496756 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.496735    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crtj\" (UniqueName: \"kubernetes.io/projected/2a7649cf-3f88-4bab-9a76-b824325a9610-kube-api-access-7crtj\") pod \"klusterlet-addon-workmgr-7cc75787b9-rn6c4\" (UID: \"2a7649cf-3f88-4bab-9a76-b824325a9610\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:03.586508 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.586423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:03.586508 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.586481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:03.586672 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:03.586545    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 20:42:03.586672 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:03.586579    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:03.586672 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:03.586608    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm podName:13da4fd0-9ad4-4721-88fa-7555047c7b69 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:05.586592933 +0000 UTC m=+100.338698887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2rgn" (UID: "13da4fd0-9ad4-4721-88fa-7555047c7b69") : configmap "kube-rbac-proxy" not found
Aug 04 20:42:03.586672 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:03.586632    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls podName:13da4fd0-9ad4-4721-88fa-7555047c7b69 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:05.586619877 +0000 UTC m=+100.338725833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2rgn" (UID: "13da4fd0-9ad4-4721-88fa-7555047c7b69") : secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:03.630915 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.630881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:03.758651 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:03.758499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"]
Aug 04 20:42:03.761214 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:03.761184    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7649cf_3f88_4bab_9a76_b824325a9610.slice/crio-3ac77f4d400974a9fb7c1a482f6035d57e3806bf0d334954cf57277015e77efe WatchSource:0}: Error finding container 3ac77f4d400974a9fb7c1a482f6035d57e3806bf0d334954cf57277015e77efe: Status 404 returned error can't find the container with id 3ac77f4d400974a9fb7c1a482f6035d57e3806bf0d334954cf57277015e77efe
Aug 04 20:42:04.045978 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.045956    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qn2gs_5302fdd4-a446-4010-8115-a4ac2915fcc6/dns-node-resolver/0.log"
Aug 04 20:42:04.186658 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.186621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4" event={"ID":"2a7649cf-3f88-4bab-9a76-b824325a9610","Type":"ContainerStarted","Data":"3ac77f4d400974a9fb7c1a482f6035d57e3806bf0d334954cf57277015e77efe"}
Aug 04 20:42:04.233638 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.233610    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-2qcgg"]
Aug 04 20:42:04.237460 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.237442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.242113 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.242093    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Aug 04 20:42:04.242213 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.242099    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Aug 04 20:42:04.242741 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.242721    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Aug 04 20:42:04.242814 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.242728    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mfbvz\""
Aug 04 20:42:04.243176 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.243158    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Aug 04 20:42:04.248260 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.248242    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Aug 04 20:42:04.257399 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.257381    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-2qcgg"]
Aug 04 20:42:04.392707 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.392636    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6321981-4380-4a31-bd58-7b64e15d9e9a-serving-cert\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.393069 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.392711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6321981-4380-4a31-bd58-7b64e15d9e9a-config\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.393069 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.392758    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6321981-4380-4a31-bd58-7b64e15d9e9a-trusted-ca\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.393069 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.392829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phr4j\" (UniqueName: \"kubernetes.io/projected/f6321981-4380-4a31-bd58-7b64e15d9e9a-kube-api-access-phr4j\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.493405 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.493367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6321981-4380-4a31-bd58-7b64e15d9e9a-config\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.493640 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.493411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6321981-4380-4a31-bd58-7b64e15d9e9a-trusted-ca\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.493640 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.493498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls\") pod \"image-registry-5fbd494568-8vch9\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") " pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:42:04.493640 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.493539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phr4j\" (UniqueName: \"kubernetes.io/projected/f6321981-4380-4a31-bd58-7b64e15d9e9a-kube-api-access-phr4j\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.493640 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.493582    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:42:04.493640 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.493630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6321981-4380-4a31-bd58-7b64e15d9e9a-serving-cert\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.493929 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:04.493757    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 20:42:04.493929 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:04.493782    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbd494568-8vch9: secret "image-registry-tls" not found
Aug 04 20:42:04.493929 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:04.493838    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 20:42:04.493929 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:04.493841    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls podName:a8c1ad3c-f203-48b1-9368-866418f92fc5 nodeName:}" failed. No retries permitted until 2026-08-04 20:43:08.493820224 +0000 UTC m=+163.245926178 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls") pod "image-registry-5fbd494568-8vch9" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5") : secret "image-registry-tls" not found
Aug 04 20:42:04.493929 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:04.493894    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls podName:d6f2150f-4332-4f28-abe4-72de91210bfa nodeName:}" failed. No retries permitted until 2026-08-04 20:43:08.493879341 +0000 UTC m=+163.245985308 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls") pod "dns-default-zr7n9" (UID: "d6f2150f-4332-4f28-abe4-72de91210bfa") : secret "dns-default-metrics-tls" not found
Aug 04 20:42:04.494272 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.494175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6321981-4380-4a31-bd58-7b64e15d9e9a-config\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.494445 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.494396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6321981-4380-4a31-bd58-7b64e15d9e9a-trusted-ca\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.496455 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.496431    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6321981-4380-4a31-bd58-7b64e15d9e9a-serving-cert\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.507566 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.507539    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phr4j\" (UniqueName: \"kubernetes.io/projected/f6321981-4380-4a31-bd58-7b64e15d9e9a-kube-api-access-phr4j\") pod \"console-operator-5ccd6b445f-2qcgg\" (UID: \"f6321981-4380-4a31-bd58-7b64e15d9e9a\") " pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.545761 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.545731    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:04.683243 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.683170    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-2qcgg"]
Aug 04 20:42:04.686262 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:04.686236    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6321981_4380_4a31_bd58_7b64e15d9e9a.slice/crio-4637531714f34c6519a33095125f2cfc4d2fb4dadc023c6c3865f45b99d13352 WatchSource:0}: Error finding container 4637531714f34c6519a33095125f2cfc4d2fb4dadc023c6c3865f45b99d13352: Status 404 returned error can't find the container with id 4637531714f34c6519a33095125f2cfc4d2fb4dadc023c6c3865f45b99d13352
Aug 04 20:42:04.842397 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:04.842370    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xltcv_430b112c-c5d1-4736-916b-6577cdc6464a/node-ca/0.log"
Aug 04 20:42:05.192849 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:05.192777    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg" event={"ID":"f6321981-4380-4a31-bd58-7b64e15d9e9a","Type":"ContainerStarted","Data":"4637531714f34c6519a33095125f2cfc4d2fb4dadc023c6c3865f45b99d13352"}
Aug 04 20:42:05.242686 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:05.242606    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4nrkd_cbe77e45-7755-4781-93b4-75fe732d1507/serve-healthcheck-canary/0.log"
Aug 04 20:42:05.605150 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:05.604875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:05.605150 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:05.604967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:05.605150 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:05.605079    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 20:42:05.605150 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:05.605106    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:05.605150 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:05.605150    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm podName:13da4fd0-9ad4-4721-88fa-7555047c7b69 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:09.605131036 +0000 UTC m=+104.357237008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2rgn" (UID: "13da4fd0-9ad4-4721-88fa-7555047c7b69") : configmap "kube-rbac-proxy" not found
Aug 04 20:42:05.605849 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:05.605167    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls podName:13da4fd0-9ad4-4721-88fa-7555047c7b69 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:09.605158721 +0000 UTC m=+104.357264677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2rgn" (UID: "13da4fd0-9ad4-4721-88fa-7555047c7b69") : secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:08.129755 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.129724    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v7t6q"
Aug 04 20:42:08.222427 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.222395    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"]
Aug 04 20:42:08.237013 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.236988    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"]
Aug 04 20:42:08.237153 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.237086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"
Aug 04 20:42:08.240416 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.240395    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Aug 04 20:42:08.240540 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.240417    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Aug 04 20:42:08.240540 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.240418    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-72m7f\""
Aug 04 20:42:08.241046 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.240849    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Aug 04 20:42:08.241046 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.240860    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Aug 04 20:42:08.330161 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.330138    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"]
Aug 04 20:42:08.330299 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.330282    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0803eda-abb5-440a-8fcd-f4ffcb3257f6-serving-cert\") pod \"service-ca-operator-784f97bcf6-2qttv\" (UID: \"a0803eda-abb5-440a-8fcd-f4ffcb3257f6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"
Aug 04 20:42:08.330360 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.330322    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0803eda-abb5-440a-8fcd-f4ffcb3257f6-config\") pod \"service-ca-operator-784f97bcf6-2qttv\" (UID: \"a0803eda-abb5-440a-8fcd-f4ffcb3257f6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"
Aug 04 20:42:08.330418 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.330402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtx7h\" (UniqueName: \"kubernetes.io/projected/a0803eda-abb5-440a-8fcd-f4ffcb3257f6-kube-api-access-wtx7h\") pod \"service-ca-operator-784f97bcf6-2qttv\" (UID: \"a0803eda-abb5-440a-8fcd-f4ffcb3257f6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"
Aug 04 20:42:08.354296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.354274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"]
Aug 04 20:42:08.354440 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.354408    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"
Aug 04 20:42:08.358451 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.358434    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Aug 04 20:42:08.358525 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.358454    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Aug 04 20:42:08.358525 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.358463    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-98zpz\""
Aug 04 20:42:08.358525 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.358487    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Aug 04 20:42:08.359260 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.359246    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Aug 04 20:42:08.431591 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.431566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9md55\" (UniqueName: \"kubernetes.io/projected/cb97b82b-f635-4e6d-8c35-354c0e67ef49-kube-api-access-9md55\") pod \"kube-storage-version-migrator-operator-7878f4b88c-fdk5z\" (UID: \"cb97b82b-f635-4e6d-8c35-354c0e67ef49\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"
Aug 04 20:42:08.431695 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.431607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0803eda-abb5-440a-8fcd-f4ffcb3257f6-serving-cert\") pod \"service-ca-operator-784f97bcf6-2qttv\" (UID: \"a0803eda-abb5-440a-8fcd-f4ffcb3257f6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"
Aug 04 20:42:08.431695 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.431669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0803eda-abb5-440a-8fcd-f4ffcb3257f6-config\") pod \"service-ca-operator-784f97bcf6-2qttv\" (UID: \"a0803eda-abb5-440a-8fcd-f4ffcb3257f6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"
Aug 04 20:42:08.431766 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.431721    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb97b82b-f635-4e6d-8c35-354c0e67ef49-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-fdk5z\" (UID: \"cb97b82b-f635-4e6d-8c35-354c0e67ef49\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"
Aug 04 20:42:08.431870 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.431849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtx7h\" (UniqueName: \"kubernetes.io/projected/a0803eda-abb5-440a-8fcd-f4ffcb3257f6-kube-api-access-wtx7h\") pod \"service-ca-operator-784f97bcf6-2qttv\" (UID: \"a0803eda-abb5-440a-8fcd-f4ffcb3257f6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"
Aug 04 20:42:08.431950 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.431934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb97b82b-f635-4e6d-8c35-354c0e67ef49-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-fdk5z\" (UID: \"cb97b82b-f635-4e6d-8c35-354c0e67ef49\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"
Aug 04 20:42:08.432302 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.432282    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0803eda-abb5-440a-8fcd-f4ffcb3257f6-config\") pod \"service-ca-operator-784f97bcf6-2qttv\" (UID: \"a0803eda-abb5-440a-8fcd-f4ffcb3257f6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"
Aug 04 20:42:08.434080 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.434057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0803eda-abb5-440a-8fcd-f4ffcb3257f6-serving-cert\") pod \"service-ca-operator-784f97bcf6-2qttv\" (UID: \"a0803eda-abb5-440a-8fcd-f4ffcb3257f6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"
Aug 04 20:42:08.441559 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.441517    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtx7h\" (UniqueName: \"kubernetes.io/projected/a0803eda-abb5-440a-8fcd-f4ffcb3257f6-kube-api-access-wtx7h\") pod \"service-ca-operator-784f97bcf6-2qttv\" (UID: \"a0803eda-abb5-440a-8fcd-f4ffcb3257f6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"
Aug 04 20:42:08.532729 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.532700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9md55\" (UniqueName: \"kubernetes.io/projected/cb97b82b-f635-4e6d-8c35-354c0e67ef49-kube-api-access-9md55\") pod \"kube-storage-version-migrator-operator-7878f4b88c-fdk5z\" (UID: \"cb97b82b-f635-4e6d-8c35-354c0e67ef49\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"
Aug 04 20:42:08.532833 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.532772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb97b82b-f635-4e6d-8c35-354c0e67ef49-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-fdk5z\" (UID: \"cb97b82b-f635-4e6d-8c35-354c0e67ef49\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"
Aug 04 20:42:08.532833 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.532829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb97b82b-f635-4e6d-8c35-354c0e67ef49-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-fdk5z\" (UID: \"cb97b82b-f635-4e6d-8c35-354c0e67ef49\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"
Aug 04 20:42:08.533347 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.533324    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb97b82b-f635-4e6d-8c35-354c0e67ef49-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-fdk5z\" (UID: \"cb97b82b-f635-4e6d-8c35-354c0e67ef49\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"
Aug 04 20:42:08.534861 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.534842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb97b82b-f635-4e6d-8c35-354c0e67ef49-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-fdk5z\" (UID: \"cb97b82b-f635-4e6d-8c35-354c0e67ef49\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"
Aug 04 20:42:08.541288 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.541267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9md55\" (UniqueName: \"kubernetes.io/projected/cb97b82b-f635-4e6d-8c35-354c0e67ef49-kube-api-access-9md55\") pod \"kube-storage-version-migrator-operator-7878f4b88c-fdk5z\" (UID: \"cb97b82b-f635-4e6d-8c35-354c0e67ef49\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"
Aug 04 20:42:08.547170 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.547151    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"
Aug 04 20:42:08.676810 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.676742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"
Aug 04 20:42:08.684515 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.684489    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv"]
Aug 04 20:42:08.687255 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:08.687233    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0803eda_abb5_440a_8fcd_f4ffcb3257f6.slice/crio-17f391ca605221439a17febaf91ae0044c3b181261e7247040a572ca9caa572a WatchSource:0}: Error finding container 17f391ca605221439a17febaf91ae0044c3b181261e7247040a572ca9caa572a: Status 404 returned error can't find the container with id 17f391ca605221439a17febaf91ae0044c3b181261e7247040a572ca9caa572a
Aug 04 20:42:08.821834 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:08.821805    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z"]
Aug 04 20:42:08.824690 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:08.824666    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb97b82b_f635_4e6d_8c35_354c0e67ef49.slice/crio-c61a536a883a26e03b626b411df6e41061be791b7c26c942607fc5ddb8fbd4d5 WatchSource:0}: Error finding container c61a536a883a26e03b626b411df6e41061be791b7c26c942607fc5ddb8fbd4d5: Status 404 returned error can't find the container with id c61a536a883a26e03b626b411df6e41061be791b7c26c942607fc5ddb8fbd4d5
Aug 04 20:42:09.206076 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.206045    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/0.log"
Aug 04 20:42:09.206548 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.206088    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6321981-4380-4a31-bd58-7b64e15d9e9a" containerID="eede446e4f6a9c4fcecfbea3f8cc81fe8ebc21bf32096c6d3ebb798850ba7fc7" exitCode=255
Aug 04 20:42:09.206548 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.206194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg" event={"ID":"f6321981-4380-4a31-bd58-7b64e15d9e9a","Type":"ContainerDied","Data":"eede446e4f6a9c4fcecfbea3f8cc81fe8ebc21bf32096c6d3ebb798850ba7fc7"}
Aug 04 20:42:09.206548 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.206439    2565 scope.go:117] "RemoveContainer" containerID="eede446e4f6a9c4fcecfbea3f8cc81fe8ebc21bf32096c6d3ebb798850ba7fc7"
Aug 04 20:42:09.208278 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.208252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4" event={"ID":"2a7649cf-3f88-4bab-9a76-b824325a9610","Type":"ContainerStarted","Data":"fc0b468b00d0473d4a56b0742ad59821d068570fedef8090a96af03ccf78416c"}
Aug 04 20:42:09.208423 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.208372    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:09.209936 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.209912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z" event={"ID":"cb97b82b-f635-4e6d-8c35-354c0e67ef49","Type":"ContainerStarted","Data":"c61a536a883a26e03b626b411df6e41061be791b7c26c942607fc5ddb8fbd4d5"}
Aug 04 20:42:09.210475 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.210458    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4"
Aug 04 20:42:09.211186 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.211159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv" event={"ID":"a0803eda-abb5-440a-8fcd-f4ffcb3257f6","Type":"ContainerStarted","Data":"17f391ca605221439a17febaf91ae0044c3b181261e7247040a572ca9caa572a"}
Aug 04 20:42:09.244016 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.243973    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cc75787b9-rn6c4" podStartSLOduration=1.707119906 podStartE2EDuration="6.24395747s" totalImagesPullingTime="4.536837564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:09.242937188 +0000 UTC m=+103.995043166" watchObservedRunningTime="2026-08-04 20:42:09.24395747 +0000 UTC m=+103.996063492"
Aug 04 20:42:09.643770 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.643725    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:09.643955 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:09.643810    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:09.643955 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:09.643916    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 20:42:09.644043 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:09.643983    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm podName:13da4fd0-9ad4-4721-88fa-7555047c7b69 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:17.643957349 +0000 UTC m=+112.396063308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2rgn" (UID: "13da4fd0-9ad4-4721-88fa-7555047c7b69") : configmap "kube-rbac-proxy" not found
Aug 04 20:42:09.644238 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:09.644210    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:09.644334 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:09.644283    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls podName:13da4fd0-9ad4-4721-88fa-7555047c7b69 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:17.644265302 +0000 UTC m=+112.396371266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2rgn" (UID: "13da4fd0-9ad4-4721-88fa-7555047c7b69") : secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:10.011811 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.011778    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-8ctcg"]
Aug 04 20:42:10.024919 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.024896    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8ctcg"
Aug 04 20:42:10.025136 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.025097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-8ctcg"]
Aug 04 20:42:10.027207 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.027189    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6trf5\""
Aug 04 20:42:10.148403 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.148370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7lx2\" (UniqueName: \"kubernetes.io/projected/0a2ccd45-7b3b-4737-854a-3375f3c3ce0b-kube-api-access-w7lx2\") pod \"network-check-source-65bdbbff45-8ctcg\" (UID: \"0a2ccd45-7b3b-4737-854a-3375f3c3ce0b\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8ctcg"
Aug 04 20:42:10.216198 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.216168    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 20:42:10.216683 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.216661    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/0.log"
Aug 04 20:42:10.216805 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.216699    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6321981-4380-4a31-bd58-7b64e15d9e9a" containerID="1a6eada8103c9d43356355d389ab48c54cc9a2380b790c3f63669eccb641029d" exitCode=255
Aug 04 20:42:10.216805 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.216789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg" event={"ID":"f6321981-4380-4a31-bd58-7b64e15d9e9a","Type":"ContainerDied","Data":"1a6eada8103c9d43356355d389ab48c54cc9a2380b790c3f63669eccb641029d"}
Aug 04 20:42:10.216883 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.216835    2565 scope.go:117] "RemoveContainer" containerID="eede446e4f6a9c4fcecfbea3f8cc81fe8ebc21bf32096c6d3ebb798850ba7fc7"
Aug 04 20:42:10.217067 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.217031    2565 scope.go:117] "RemoveContainer" containerID="1a6eada8103c9d43356355d389ab48c54cc9a2380b790c3f63669eccb641029d"
Aug 04 20:42:10.217231 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:10.217209    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-2qcgg_openshift-console-operator(f6321981-4380-4a31-bd58-7b64e15d9e9a)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg" podUID="f6321981-4380-4a31-bd58-7b64e15d9e9a"
Aug 04 20:42:10.249242 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.249201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7lx2\" (UniqueName: \"kubernetes.io/projected/0a2ccd45-7b3b-4737-854a-3375f3c3ce0b-kube-api-access-w7lx2\") pod \"network-check-source-65bdbbff45-8ctcg\" (UID: \"0a2ccd45-7b3b-4737-854a-3375f3c3ce0b\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8ctcg"
Aug 04 20:42:10.258277 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.258254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7lx2\" (UniqueName: \"kubernetes.io/projected/0a2ccd45-7b3b-4737-854a-3375f3c3ce0b-kube-api-access-w7lx2\") pod \"network-check-source-65bdbbff45-8ctcg\" (UID: \"0a2ccd45-7b3b-4737-854a-3375f3c3ce0b\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8ctcg"
Aug 04 20:42:10.333936 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:10.333858    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8ctcg"
Aug 04 20:42:11.221224 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:11.221038    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 20:42:11.221612 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:11.221336    2565 scope.go:117] "RemoveContainer" containerID="1a6eada8103c9d43356355d389ab48c54cc9a2380b790c3f63669eccb641029d"
Aug 04 20:42:11.221612 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:11.221509    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-2qcgg_openshift-console-operator(f6321981-4380-4a31-bd58-7b64e15d9e9a)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg" podUID="f6321981-4380-4a31-bd58-7b64e15d9e9a"
Aug 04 20:42:11.282424 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:11.281370    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-8ctcg"]
Aug 04 20:42:11.285216 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:11.285182    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2ccd45_7b3b_4737_854a_3375f3c3ce0b.slice/crio-8e3ccd40e4e9ceaa147d2bf8b9321d6455a5f60927957948ad1508b08bcc1297 WatchSource:0}: Error finding container 8e3ccd40e4e9ceaa147d2bf8b9321d6455a5f60927957948ad1508b08bcc1297: Status 404 returned error can't find the container with id 8e3ccd40e4e9ceaa147d2bf8b9321d6455a5f60927957948ad1508b08bcc1297
Aug 04 20:42:12.230834 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:12.230791    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8ctcg" event={"ID":"0a2ccd45-7b3b-4737-854a-3375f3c3ce0b","Type":"ContainerStarted","Data":"be5a88851d1537954b9a082db40bab0cfa579a1ad213edfc311564dc7b959bc9"}
Aug 04 20:42:12.231441 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:12.230883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8ctcg" event={"ID":"0a2ccd45-7b3b-4737-854a-3375f3c3ce0b","Type":"ContainerStarted","Data":"8e3ccd40e4e9ceaa147d2bf8b9321d6455a5f60927957948ad1508b08bcc1297"}
Aug 04 20:42:12.232327 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:12.232293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z" event={"ID":"cb97b82b-f635-4e6d-8c35-354c0e67ef49","Type":"ContainerStarted","Data":"9b38ed6c5a7f04d4fe423d757d9092e387a22baeae5eb85eb4c64a73671c3b8e"}
Aug 04 20:42:12.233835 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:12.233792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv" event={"ID":"a0803eda-abb5-440a-8fcd-f4ffcb3257f6","Type":"ContainerStarted","Data":"d6e22f010f477559b79b9b7d9767f1e9e74d514f957560c9eec234203d25e154"}
Aug 04 20:42:12.248025 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:12.247983    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8ctcg" podStartSLOduration=3.247971034 podStartE2EDuration="3.247971034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:12.247167897 +0000 UTC m=+106.999273874" watchObservedRunningTime="2026-08-04 20:42:12.247971034 +0000 UTC m=+107.000076989"
Aug 04 20:42:12.267100 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:12.267062    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv" podStartSLOduration=1.800098663 podStartE2EDuration="4.267050294s" totalImagesPullingTime="2.466951631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:12.266304763 +0000 UTC m=+107.018410743" watchObservedRunningTime="2026-08-04 20:42:12.267050294 +0000 UTC m=+107.019156271"
Aug 04 20:42:12.282194 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:12.282151    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z" podStartSLOduration=1.950728985 podStartE2EDuration="4.282135598s" totalImagesPullingTime="2.331406613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:12.281525674 +0000 UTC m=+107.033631650" watchObservedRunningTime="2026-08-04 20:42:12.282135598 +0000 UTC m=+107.034241577"
Aug 04 20:42:14.546153 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.546119    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:14.546530 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.546173    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:14.546530 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.546517    2565 scope.go:117] "RemoveContainer" containerID="1a6eada8103c9d43356355d389ab48c54cc9a2380b790c3f63669eccb641029d"
Aug 04 20:42:14.546673 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:14.546654    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-2qcgg_openshift-console-operator(f6321981-4380-4a31-bd58-7b64e15d9e9a)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg" podUID="f6321981-4380-4a31-bd58-7b64e15d9e9a"
Aug 04 20:42:14.593752 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.593722    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-wdkvk"]
Aug 04 20:42:14.597724 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.597705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk"
Aug 04 20:42:14.600799 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.600776    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Aug 04 20:42:14.601587 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.601565    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Aug 04 20:42:14.601681 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.601569    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qbg59\""
Aug 04 20:42:14.601681 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.601569    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Aug 04 20:42:14.601681 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.601640    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Aug 04 20:42:14.610498 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.610477    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-wdkvk"]
Aug 04 20:42:14.690572 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.690547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72p7l\" (UniqueName: \"kubernetes.io/projected/c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0-kube-api-access-72p7l\") pod \"service-ca-98c88fbb5-wdkvk\" (UID: \"c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk"
Aug 04 20:42:14.690712 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.690689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0-signing-key\") pod \"service-ca-98c88fbb5-wdkvk\" (UID: \"c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk"
Aug 04 20:42:14.690854 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.690830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0-signing-cabundle\") pod \"service-ca-98c88fbb5-wdkvk\" (UID: \"c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk"
Aug 04 20:42:14.791543 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.791515    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72p7l\" (UniqueName: \"kubernetes.io/projected/c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0-kube-api-access-72p7l\") pod \"service-ca-98c88fbb5-wdkvk\" (UID: \"c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk"
Aug 04 20:42:14.791643 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.791569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0-signing-key\") pod \"service-ca-98c88fbb5-wdkvk\" (UID: \"c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk"
Aug 04 20:42:14.791686 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.791645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0-signing-cabundle\") pod \"service-ca-98c88fbb5-wdkvk\" (UID: \"c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk"
Aug 04 20:42:14.792262 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.792244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0-signing-cabundle\") pod \"service-ca-98c88fbb5-wdkvk\" (UID: \"c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk"
Aug 04 20:42:14.793946 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.793924    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0-signing-key\") pod \"service-ca-98c88fbb5-wdkvk\" (UID: \"c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk"
Aug 04 20:42:14.800310 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.800241    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72p7l\" (UniqueName: \"kubernetes.io/projected/c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0-kube-api-access-72p7l\") pod \"service-ca-98c88fbb5-wdkvk\" (UID: \"c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk"
Aug 04 20:42:14.906623 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:14.906593    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk"
Aug 04 20:42:15.029318 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:15.029289    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-wdkvk"]
Aug 04 20:42:15.032519 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:15.032492    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c18fd1_3f42_47f9_8c65_e7ebab7c7bf0.slice/crio-8097f8cfc23f72a1cdbafe7192206fa9ebea07a696e271bb5110661d407a5a33 WatchSource:0}: Error finding container 8097f8cfc23f72a1cdbafe7192206fa9ebea07a696e271bb5110661d407a5a33: Status 404 returned error can't find the container with id 8097f8cfc23f72a1cdbafe7192206fa9ebea07a696e271bb5110661d407a5a33
Aug 04 20:42:15.244775 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:15.244741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk" event={"ID":"c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0","Type":"ContainerStarted","Data":"1bbf5936a256f5f6d713604a76d8565eefdc38f33856ce581a5dcbc72beeb22b"}
Aug 04 20:42:15.244926 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:15.244787    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk" event={"ID":"c0c18fd1-3f42-47f9-8c65-e7ebab7c7bf0","Type":"ContainerStarted","Data":"8097f8cfc23f72a1cdbafe7192206fa9ebea07a696e271bb5110661d407a5a33"}
Aug 04 20:42:15.271204 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:15.271158    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-wdkvk" podStartSLOduration=1.271143195 podStartE2EDuration="1.271143195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:15.270563246 +0000 UTC m=+110.022669222" watchObservedRunningTime="2026-08-04 20:42:15.271143195 +0000 UTC m=+110.023249172"
Aug 04 20:42:17.716210 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:17.716170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:17.716607 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:17.716255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:17.716607 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:17.716326    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:17.716607 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:17.716419    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls podName:13da4fd0-9ad4-4721-88fa-7555047c7b69 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:33.716399445 +0000 UTC m=+128.468505400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2rgn" (UID: "13da4fd0-9ad4-4721-88fa-7555047c7b69") : secret "insights-runtime-extractor-tls" not found
Aug 04 20:42:17.716879 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:17.716860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13da4fd0-9ad4-4721-88fa-7555047c7b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:25.755999 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:25.755971    2565 scope.go:117] "RemoveContainer" containerID="1a6eada8103c9d43356355d389ab48c54cc9a2380b790c3f63669eccb641029d"
Aug 04 20:42:26.283705 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:26.283667    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 20:42:26.283863 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:26.283745    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg" event={"ID":"f6321981-4380-4a31-bd58-7b64e15d9e9a","Type":"ContainerStarted","Data":"d8bf233df3bb242c58af57d3e5e5a2fe919d7501333410c78cf5725a8243c8a0"}
Aug 04 20:42:26.284113 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:26.284088    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:26.313007 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:26.312958    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg" podStartSLOduration=18.716331995 podStartE2EDuration="22.312943724s" totalImagesPullingTime="3.596611729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:26.312786592 +0000 UTC m=+121.064892568" watchObservedRunningTime="2026-08-04 20:42:26.312943724 +0000 UTC m=+121.065049700"
Aug 04 20:42:26.973449 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:26.973418    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-2qcgg"
Aug 04 20:42:33.738125 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:33.738089    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:33.740286 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:33.740262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13da4fd0-9ad4-4721-88fa-7555047c7b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2rgn\" (UID: \"13da4fd0-9ad4-4721-88fa-7555047c7b69\") " pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:34.017277 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:34.017207    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d7shv\""
Aug 04 20:42:34.024550 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:34.024533    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n2rgn"
Aug 04 20:42:34.146618 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:34.146485    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n2rgn"]
Aug 04 20:42:34.149037 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:34.149014    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13da4fd0_9ad4_4721_88fa_7555047c7b69.slice/crio-f0b6e821f0c2723ee492a9c2abf8fbbd4b5d5904f660234503d5b3be0eb37b40 WatchSource:0}: Error finding container f0b6e821f0c2723ee492a9c2abf8fbbd4b5d5904f660234503d5b3be0eb37b40: Status 404 returned error can't find the container with id f0b6e821f0c2723ee492a9c2abf8fbbd4b5d5904f660234503d5b3be0eb37b40
Aug 04 20:42:34.307725 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:34.307648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2rgn" event={"ID":"13da4fd0-9ad4-4721-88fa-7555047c7b69","Type":"ContainerStarted","Data":"dba132eb7fc6d44082a83caf7cec88e03e69cfdfba5075b4376cd82128aa7b55"}
Aug 04 20:42:34.307725 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:34.307712    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2rgn" event={"ID":"13da4fd0-9ad4-4721-88fa-7555047c7b69","Type":"ContainerStarted","Data":"f0b6e821f0c2723ee492a9c2abf8fbbd4b5d5904f660234503d5b3be0eb37b40"}
Aug 04 20:42:35.312098 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:35.312065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2rgn" event={"ID":"13da4fd0-9ad4-4721-88fa-7555047c7b69","Type":"ContainerStarted","Data":"26d03e1593373d58021c87aaee796e3ac47274e14bf1f611ba204e3f6ebe2f24"}
Aug 04 20:42:35.552650 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:35.552612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:42:35.555255 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:35.555229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371d1e66-d185-408c-96e4-e955fb45f77d-metrics-certs\") pod \"network-metrics-daemon-744vd\" (UID: \"371d1e66-d185-408c-96e4-e955fb45f77d\") " pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:42:35.575193 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:35.575127    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h8x4x\""
Aug 04 20:42:35.582609 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:35.582589    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-744vd"
Aug 04 20:42:35.731211 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:35.731176    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-744vd"]
Aug 04 20:42:35.734547 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:35.734523    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371d1e66_d185_408c_96e4_e955fb45f77d.slice/crio-9683f11e7b0012b7893d46f89bcbf865cfbd6e5d2f13ed9036c28b6228cb6112 WatchSource:0}: Error finding container 9683f11e7b0012b7893d46f89bcbf865cfbd6e5d2f13ed9036c28b6228cb6112: Status 404 returned error can't find the container with id 9683f11e7b0012b7893d46f89bcbf865cfbd6e5d2f13ed9036c28b6228cb6112
Aug 04 20:42:36.318512 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:36.318475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2rgn" event={"ID":"13da4fd0-9ad4-4721-88fa-7555047c7b69","Type":"ContainerStarted","Data":"618c80e8fe82b22083c5da6d04f419721ffe4e910092fe4d7fd67fe5ff8fdb4e"}
Aug 04 20:42:36.320030 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:36.320006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-744vd" event={"ID":"371d1e66-d185-408c-96e4-e955fb45f77d","Type":"ContainerStarted","Data":"9683f11e7b0012b7893d46f89bcbf865cfbd6e5d2f13ed9036c28b6228cb6112"}
Aug 04 20:42:36.343137 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:36.343085    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n2rgn" podStartSLOduration=33.426343028 podStartE2EDuration="35.343069298s" totalImagesPullingTime="1.91672627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:42:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:36.341771579 +0000 UTC m=+131.093877557" watchObservedRunningTime="2026-08-04 20:42:36.343069298 +0000 UTC m=+131.095175275"
Aug 04 20:42:37.328316 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:37.328284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-744vd" event={"ID":"371d1e66-d185-408c-96e4-e955fb45f77d","Type":"ContainerStarted","Data":"6d98945ed0966a27b1fb0c01de91c60ab61337f58768e21ce1d040e190fa58c2"}
Aug 04 20:42:37.328677 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:37.328370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-744vd" event={"ID":"371d1e66-d185-408c-96e4-e955fb45f77d","Type":"ContainerStarted","Data":"a9ca6c21c0ba0975fff67acda277cc294920cc6ff2d22d2fdb8eac1b868cdfa2"}
Aug 04 20:42:37.346162 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:37.346116    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-744vd" podStartSLOduration=130.327502609 podStartE2EDuration="2m11.346102248s" totalImagesPullingTime="1.018599639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:37.345525685 +0000 UTC m=+132.097631663" watchObservedRunningTime="2026-08-04 20:42:37.346102248 +0000 UTC m=+132.098208225"
Aug 04 20:42:38.613677 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.613640    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c"]
Aug 04 20:42:38.646909 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.646865    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c"
Aug 04 20:42:38.650519 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.650489    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c"]
Aug 04 20:42:38.650713 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.650687    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Aug 04 20:42:38.650713 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.650611    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dlkjg\""
Aug 04 20:42:38.651801 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.651776    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Aug 04 20:42:38.686058 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.686027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/adab2b9e-e2af-4560-91a9-1e996c22de08-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dxf6c\" (UID: \"adab2b9e-e2af-4560-91a9-1e996c22de08\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c"
Aug 04 20:42:38.686058 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.686058    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/adab2b9e-e2af-4560-91a9-1e996c22de08-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-dxf6c\" (UID: \"adab2b9e-e2af-4560-91a9-1e996c22de08\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c"
Aug 04 20:42:38.705158 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.705128    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-qjv89"]
Aug 04 20:42:38.726265 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.726236    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9"]
Aug 04 20:42:38.726444 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.726386    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-qjv89"
Aug 04 20:42:38.731294 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.731030    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Aug 04 20:42:38.731294 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.731094    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Aug 04 20:42:38.731294 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.731100    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ntn4p\""
Aug 04 20:42:38.746333 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.746307    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9"]
Aug 04 20:42:38.746506 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.746374    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-qjv89"]
Aug 04 20:42:38.746506 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.746467    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9"
Aug 04 20:42:38.749477 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.749457    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Aug 04 20:42:38.749590 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.749459    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7bh99\""
Aug 04 20:42:38.787476 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.787441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btztf\" (UniqueName: \"kubernetes.io/projected/a86e1e49-9816-4c01-81a3-6e4850052b95-kube-api-access-btztf\") pod \"downloads-77c974ffcf-qjv89\" (UID: \"a86e1e49-9816-4c01-81a3-6e4850052b95\") " pod="openshift-console/downloads-77c974ffcf-qjv89"
Aug 04 20:42:38.787476 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.787477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/32c9759d-81c2-4f40-943c-eed9b9121f8b-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-7dhc9\" (UID: \"32c9759d-81c2-4f40-943c-eed9b9121f8b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9"
Aug 04 20:42:38.787678 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.787512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/adab2b9e-e2af-4560-91a9-1e996c22de08-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dxf6c\" (UID: \"adab2b9e-e2af-4560-91a9-1e996c22de08\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c"
Aug 04 20:42:38.787678 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.787533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/adab2b9e-e2af-4560-91a9-1e996c22de08-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-dxf6c\" (UID: \"adab2b9e-e2af-4560-91a9-1e996c22de08\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c"
Aug 04 20:42:38.788093 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.788077    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/adab2b9e-e2af-4560-91a9-1e996c22de08-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-dxf6c\" (UID: \"adab2b9e-e2af-4560-91a9-1e996c22de08\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c"
Aug 04 20:42:38.789942 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.789919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/adab2b9e-e2af-4560-91a9-1e996c22de08-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dxf6c\" (UID: \"adab2b9e-e2af-4560-91a9-1e996c22de08\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c"
Aug 04 20:42:38.888663 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.888569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btztf\" (UniqueName: \"kubernetes.io/projected/a86e1e49-9816-4c01-81a3-6e4850052b95-kube-api-access-btztf\") pod \"downloads-77c974ffcf-qjv89\" (UID: \"a86e1e49-9816-4c01-81a3-6e4850052b95\") " pod="openshift-console/downloads-77c974ffcf-qjv89"
Aug 04 20:42:38.888663 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.888607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/32c9759d-81c2-4f40-943c-eed9b9121f8b-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-7dhc9\" (UID: \"32c9759d-81c2-4f40-943c-eed9b9121f8b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9"
Aug 04 20:42:38.891046 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.891025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/32c9759d-81c2-4f40-943c-eed9b9121f8b-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-7dhc9\" (UID: \"32c9759d-81c2-4f40-943c-eed9b9121f8b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9"
Aug 04 20:42:38.902029 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.902001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btztf\" (UniqueName: \"kubernetes.io/projected/a86e1e49-9816-4c01-81a3-6e4850052b95-kube-api-access-btztf\") pod \"downloads-77c974ffcf-qjv89\" (UID: \"a86e1e49-9816-4c01-81a3-6e4850052b95\") " pod="openshift-console/downloads-77c974ffcf-qjv89"
Aug 04 20:42:38.958805 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:38.958778    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c"
Aug 04 20:42:39.036715 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:39.036685    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-qjv89"
Aug 04 20:42:39.054872 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:39.054844    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9"
Aug 04 20:42:39.133052 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:39.132923    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c"]
Aug 04 20:42:39.137017 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:39.136987    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadab2b9e_e2af_4560_91a9_1e996c22de08.slice/crio-f37498c3a6077bba54e167ea7b038f4c4f33c7b2cd1279b07f79fd60fd8d76f1 WatchSource:0}: Error finding container f37498c3a6077bba54e167ea7b038f4c4f33c7b2cd1279b07f79fd60fd8d76f1: Status 404 returned error can't find the container with id f37498c3a6077bba54e167ea7b038f4c4f33c7b2cd1279b07f79fd60fd8d76f1
Aug 04 20:42:39.208989 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:39.208961    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-qjv89"]
Aug 04 20:42:39.210334 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:39.210309    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86e1e49_9816_4c01_81a3_6e4850052b95.slice/crio-ccfce84ccd290d83d544f1602bc2dc7a288d37976bf7ad9d3c399d4108381599 WatchSource:0}: Error finding container ccfce84ccd290d83d544f1602bc2dc7a288d37976bf7ad9d3c399d4108381599: Status 404 returned error can't find the container with id ccfce84ccd290d83d544f1602bc2dc7a288d37976bf7ad9d3c399d4108381599
Aug 04 20:42:39.225514 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:39.225498    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9"]
Aug 04 20:42:39.227863 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:39.227838    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c9759d_81c2_4f40_943c_eed9b9121f8b.slice/crio-7f1ffc479ca64fa8cc63f43f4fa0cf9fc5978c3b570b27c2c4080d146e163c26 WatchSource:0}: Error finding container 7f1ffc479ca64fa8cc63f43f4fa0cf9fc5978c3b570b27c2c4080d146e163c26: Status 404 returned error can't find the container with id 7f1ffc479ca64fa8cc63f43f4fa0cf9fc5978c3b570b27c2c4080d146e163c26
Aug 04 20:42:39.335180 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:39.335141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c" event={"ID":"adab2b9e-e2af-4560-91a9-1e996c22de08","Type":"ContainerStarted","Data":"f37498c3a6077bba54e167ea7b038f4c4f33c7b2cd1279b07f79fd60fd8d76f1"}
Aug 04 20:42:39.336298 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:39.336269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9" event={"ID":"32c9759d-81c2-4f40-943c-eed9b9121f8b","Type":"ContainerStarted","Data":"7f1ffc479ca64fa8cc63f43f4fa0cf9fc5978c3b570b27c2c4080d146e163c26"}
Aug 04 20:42:39.337309 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:39.337285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-qjv89" event={"ID":"a86e1e49-9816-4c01-81a3-6e4850052b95","Type":"ContainerStarted","Data":"ccfce84ccd290d83d544f1602bc2dc7a288d37976bf7ad9d3c399d4108381599"}
Aug 04 20:42:41.347747 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:41.347706    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c" event={"ID":"adab2b9e-e2af-4560-91a9-1e996c22de08","Type":"ContainerStarted","Data":"0f9c21d922a2b1fecb567e79f0678362e2e62829af59ffc2d500701f92c3e4d1"}
Aug 04 20:42:41.349268 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:41.349243    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9" event={"ID":"32c9759d-81c2-4f40-943c-eed9b9121f8b","Type":"ContainerStarted","Data":"fcd03febb27fc2c6028b72b1a076ef049f366f326e777b9c727c701446f4ebec"}
Aug 04 20:42:41.349417 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:41.349292    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9"
Aug 04 20:42:41.355054 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:41.355030    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9"
Aug 04 20:42:41.365564 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:41.365518    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-dxf6c" podStartSLOduration=1.8980588059999999 podStartE2EDuration="3.365502727s" totalImagesPullingTime="1.467443921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:41.364812366 +0000 UTC m=+136.116918375" watchObservedRunningTime="2026-08-04 20:42:41.365502727 +0000 UTC m=+136.117608707"
Aug 04 20:42:41.383171 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:41.383124    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7dhc9" podStartSLOduration=2.003354802 podStartE2EDuration="3.383109104s" totalImagesPullingTime="1.379754302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:41.382604271 +0000 UTC m=+136.134710263" watchObservedRunningTime="2026-08-04 20:42:41.383109104 +0000 UTC m=+136.135215059"
Aug 04 20:42:46.697886 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.697846    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"]
Aug 04 20:42:46.701457 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.701435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.706513 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.705312    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Aug 04 20:42:46.706513 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.705859    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 04 20:42:46.706513 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.706275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Aug 04 20:42:46.706741 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.706651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-t5xbq\""
Aug 04 20:42:46.708375 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.707872    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Aug 04 20:42:46.711023 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.710247    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 04 20:42:46.711023 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.710513    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 04 20:42:46.716604 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.716580    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pp75k"]
Aug 04 20:42:46.720154 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.720130    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.723592 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.723573    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cjqnv\""
Aug 04 20:42:46.723750 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.723712    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 04 20:42:46.725594 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.725560    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 04 20:42:46.726195 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.726176    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 04 20:42:46.729663 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.729646    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"]
Aug 04 20:42:46.858853 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.858818    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-accelerators-collector-config\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.859043 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.858869    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.859043 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.858900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.859043 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.858969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-wtmp\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.859043 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.858998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6wv\" (UniqueName: \"kubernetes.io/projected/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-api-access-vc6wv\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.859043 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.859041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74c4817c-585c-48c7-82f9-95f4a0b1b958-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.859296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.859066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-textfile\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.859296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.859089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-tls\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.859296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.859126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b59f4a25-b7f3-41c4-b839-2b3b2a906294-metrics-client-ca\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.859296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.859197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b59f4a25-b7f3-41c4-b839-2b3b2a906294-sys\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.859296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.859222    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xhhh\" (UniqueName: \"kubernetes.io/projected/b59f4a25-b7f3-41c4-b839-2b3b2a906294-kube-api-access-4xhhh\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.859296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.859254    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.859296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.859288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74c4817c-585c-48c7-82f9-95f4a0b1b958-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.859709 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.859335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b59f4a25-b7f3-41c4-b839-2b3b2a906294-root\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.859709 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.859413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.960158 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960077    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-tls\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.960158 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b59f4a25-b7f3-41c4-b839-2b3b2a906294-metrics-client-ca\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.960158 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b59f4a25-b7f3-41c4-b839-2b3b2a906294-sys\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.960453 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xhhh\" (UniqueName: \"kubernetes.io/projected/b59f4a25-b7f3-41c4-b839-2b3b2a906294-kube-api-access-4xhhh\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.960453 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.960453 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74c4817c-585c-48c7-82f9-95f4a0b1b958-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.960453 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b59f4a25-b7f3-41c4-b839-2b3b2a906294-root\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.960453 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:46.960248    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Aug 04 20:42:46.960453 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960264    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.960453 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-accelerators-collector-config\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.960453 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:46.960314    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-tls podName:b59f4a25-b7f3-41c4-b839-2b3b2a906294 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:47.460294234 +0000 UTC m=+142.212400195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-tls") pod "node-exporter-pp75k" (UID: "b59f4a25-b7f3-41c4-b839-2b3b2a906294") : secret "node-exporter-tls" not found
Aug 04 20:42:46.960453 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.960453 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.960453 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960444    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-wtmp\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.960981 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6wv\" (UniqueName: \"kubernetes.io/projected/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-api-access-vc6wv\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.960981 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960513    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74c4817c-585c-48c7-82f9-95f4a0b1b958-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.960981 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-textfile\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.960981 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-accelerators-collector-config\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.960981 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.960837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-textfile\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.961216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.961045    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b59f4a25-b7f3-41c4-b839-2b3b2a906294-metrics-client-ca\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.961216 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.961083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b59f4a25-b7f3-41c4-b839-2b3b2a906294-sys\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.961647 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.961434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b59f4a25-b7f3-41c4-b839-2b3b2a906294-root\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.961647 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.961466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-wtmp\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.961647 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:46.961555    2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Aug 04 20:42:46.961647 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:46.961612    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-tls podName:74c4817c-585c-48c7-82f9-95f4a0b1b958 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:47.461596004 +0000 UTC m=+142.213701961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-dkgh8" (UID: "74c4817c-585c-48c7-82f9-95f4a0b1b958") : secret "kube-state-metrics-tls" not found
Aug 04 20:42:46.961924 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.961773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74c4817c-585c-48c7-82f9-95f4a0b1b958-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.961924 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.961853    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.962037 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.961956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74c4817c-585c-48c7-82f9-95f4a0b1b958-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.964555 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.964445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:46.964555 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.964517    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.970795 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.970755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xhhh\" (UniqueName: \"kubernetes.io/projected/b59f4a25-b7f3-41c4-b839-2b3b2a906294-kube-api-access-4xhhh\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:46.971151 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:46.971118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6wv\" (UniqueName: \"kubernetes.io/projected/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-api-access-vc6wv\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:47.465417 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.465384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-tls\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:47.465583 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.465492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:47.467961 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.467933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b59f4a25-b7f3-41c4-b839-2b3b2a906294-node-exporter-tls\") pod \"node-exporter-pp75k\" (UID: \"b59f4a25-b7f3-41c4-b839-2b3b2a906294\") " pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:47.468416 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.468395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74c4817c-585c-48c7-82f9-95f4a0b1b958-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-dkgh8\" (UID: \"74c4817c-585c-48c7-82f9-95f4a0b1b958\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:47.617723 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.617688    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"
Aug 04 20:42:47.632525 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.632494    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pp75k"
Aug 04 20:42:47.650399 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:47.650363    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59f4a25_b7f3_41c4_b839_2b3b2a906294.slice/crio-02b536678ddca0633ff4c438fbcc78ed9482598e97ad3987941c611f576ddd98 WatchSource:0}: Error finding container 02b536678ddca0633ff4c438fbcc78ed9482598e97ad3987941c611f576ddd98: Status 404 returned error can't find the container with id 02b536678ddca0633ff4c438fbcc78ed9482598e97ad3987941c611f576ddd98
Aug 04 20:42:47.761718 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.761688    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 20:42:47.766768 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.766748    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.769223 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.769197    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Aug 04 20:42:47.769420 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.769400    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Aug 04 20:42:47.770017 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.769995    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Aug 04 20:42:47.770110 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.770040    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Aug 04 20:42:47.770165 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.770001    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Aug 04 20:42:47.770480 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.770222    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-87kjp\""
Aug 04 20:42:47.770480 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.770376    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Aug 04 20:42:47.770480 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.770383    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Aug 04 20:42:47.770480 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.770414    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Aug 04 20:42:47.770738 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.770593    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Aug 04 20:42:47.776457 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.776257    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8"]
Aug 04 20:42:47.781901 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.781878    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 20:42:47.782809 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:47.782780    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c4817c_585c_48c7_82f9_95f4a0b1b958.slice/crio-a823a800f3e3d3f0cf14c668c1e311fbde9bba47192721c499513a0cdc5994af WatchSource:0}: Error finding container a823a800f3e3d3f0cf14c668c1e311fbde9bba47192721c499513a0cdc5994af: Status 404 returned error can't find the container with id a823a800f3e3d3f0cf14c668c1e311fbde9bba47192721c499513a0cdc5994af
Aug 04 20:42:47.869660 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.869627    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-web-config\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.869660 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.869666    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-out\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.869862 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.869692    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.869862 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.869792    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.869862 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.869825    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.870016 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.869873    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.870016 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.869929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.870016 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.869994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.870159 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.870023    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.870159 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.870063    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.870159 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.870099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74kv\" (UniqueName: \"kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-kube-api-access-b74kv\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.870159 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.870144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-volume\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.870369 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.870217    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971365 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971322    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971519 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971519 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971519 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74kv\" (UniqueName: \"kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-kube-api-access-b74kv\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971519 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-volume\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971519 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:47.971495    2565 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Aug 04 20:42:47.971519 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971848 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:47.971572    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-main-tls podName:da807b0b-ed13-4af1-967d-c79cd5d5b058 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:48.471551529 +0000 UTC m=+143.223657501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058") : secret "alertmanager-main-tls" not found
Aug 04 20:42:47.971848 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-web-config\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971848 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-out\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971848 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971848 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971848 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971738    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.971848 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.971792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.972642 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.972612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.973822 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.973797    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.974366 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.974190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.974551 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.974528    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.976023 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.975947    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.976257 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.976234    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.976738 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.976714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-web-config\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.976944 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.976898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.977140 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.977116    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.977292 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.977255    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-out\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.977444 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.977424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-volume\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.978717 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.978695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:47.983037 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:47.983015    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74kv\" (UniqueName: \"kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-kube-api-access-b74kv\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:48.379892 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:48.379758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8" event={"ID":"74c4817c-585c-48c7-82f9-95f4a0b1b958","Type":"ContainerStarted","Data":"a823a800f3e3d3f0cf14c668c1e311fbde9bba47192721c499513a0cdc5994af"}
Aug 04 20:42:48.381726 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:48.381652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pp75k" event={"ID":"b59f4a25-b7f3-41c4-b839-2b3b2a906294","Type":"ContainerStarted","Data":"02b536678ddca0633ff4c438fbcc78ed9482598e97ad3987941c611f576ddd98"}
Aug 04 20:42:48.479816 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:48.479724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:48.483973 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:48.483917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:48.679683 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:48.679606    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:42:51.440569 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:51.440537    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"]
Aug 04 20:42:51.443916 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:51.443892    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"
Aug 04 20:42:51.446548 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:51.446526    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Aug 04 20:42:51.446668 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:51.446626    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xd7d2\""
Aug 04 20:42:51.452955 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:51.452861    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"]
Aug 04 20:42:51.607941 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:51.607905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/75b9271e-d59d-4700-a5c5-886da39c25d4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-jg6db\" (UID: \"75b9271e-d59d-4700-a5c5-886da39c25d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"
Aug 04 20:42:51.608131 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:51.607972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jzck\" (UniqueName: \"kubernetes.io/projected/75b9271e-d59d-4700-a5c5-886da39c25d4-kube-api-access-8jzck\") pod \"monitoring-plugin-6dffbbd67-jg6db\" (UID: \"75b9271e-d59d-4700-a5c5-886da39c25d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"
Aug 04 20:42:51.708730 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:51.708652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/75b9271e-d59d-4700-a5c5-886da39c25d4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-jg6db\" (UID: \"75b9271e-d59d-4700-a5c5-886da39c25d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"
Aug 04 20:42:51.708730 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:51.708722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jzck\" (UniqueName: \"kubernetes.io/projected/75b9271e-d59d-4700-a5c5-886da39c25d4-kube-api-access-8jzck\") pod \"monitoring-plugin-6dffbbd67-jg6db\" (UID: \"75b9271e-d59d-4700-a5c5-886da39c25d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"
Aug 04 20:42:51.708971 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:51.708812    2565 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Aug 04 20:42:51.708971 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:42:51.708879    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b9271e-d59d-4700-a5c5-886da39c25d4-monitoring-plugin-cert podName:75b9271e-d59d-4700-a5c5-886da39c25d4 nodeName:}" failed. No retries permitted until 2026-08-04 20:42:52.208860325 +0000 UTC m=+146.960966296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/75b9271e-d59d-4700-a5c5-886da39c25d4-monitoring-plugin-cert") pod "monitoring-plugin-6dffbbd67-jg6db" (UID: "75b9271e-d59d-4700-a5c5-886da39c25d4") : secret "monitoring-plugin-cert" not found
Aug 04 20:42:51.718133 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:51.718101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jzck\" (UniqueName: \"kubernetes.io/projected/75b9271e-d59d-4700-a5c5-886da39c25d4-kube-api-access-8jzck\") pod \"monitoring-plugin-6dffbbd67-jg6db\" (UID: \"75b9271e-d59d-4700-a5c5-886da39c25d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"
Aug 04 20:42:52.213334 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.213295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/75b9271e-d59d-4700-a5c5-886da39c25d4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-jg6db\" (UID: \"75b9271e-d59d-4700-a5c5-886da39c25d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"
Aug 04 20:42:52.216016 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.215983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/75b9271e-d59d-4700-a5c5-886da39c25d4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-jg6db\" (UID: \"75b9271e-d59d-4700-a5c5-886da39c25d4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"
Aug 04 20:42:52.355762 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.355730    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"
Aug 04 20:42:52.887251 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.887218    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 20:42:52.891399 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.891372    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:52.894122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.894058    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Aug 04 20:42:52.894333 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.894133    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Aug 04 20:42:52.894333 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.894134    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Aug 04 20:42:52.894695 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.894674    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Aug 04 20:42:52.894800 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.894719    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Aug 04 20:42:52.894800 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.894742    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8p8xp\""
Aug 04 20:42:52.894800 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.894757    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-17o59lmt02unb\""
Aug 04 20:42:52.895143 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.895063    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Aug 04 20:42:52.895143 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.895140    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Aug 04 20:42:52.895314 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.895157    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Aug 04 20:42:52.895314 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.895278    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Aug 04 20:42:52.895442 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.895391    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Aug 04 20:42:52.895497 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.895391    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Aug 04 20:42:52.895604 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.895570    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Aug 04 20:42:52.895829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.895808    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Aug 04 20:42:52.902908 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.900692    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Aug 04 20:42:52.903388 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.903366    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Aug 04 20:42:52.911451 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:52.911428    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 20:42:53.021726 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.021687    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-config\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.021917 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.021779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.021917 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.021849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.021917 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.021880    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.021917 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.021909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.021933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t58n\" (UniqueName: \"kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-kube-api-access-7t58n\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.021962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022072    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-config-out\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022244    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022313    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-web-config\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.022464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.022453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123214 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123235    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-config\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t58n\" (UniqueName: \"kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-kube-api-access-7t58n\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.123813 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.124063 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.123933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.125132 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.124581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-config-out\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.125132 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.124635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.125132 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.124680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.125132 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.124730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.125132 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.124761    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.125132 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.124804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.125132 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.124847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-web-config\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.126475 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.126384    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.126583 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.126560    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-config\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.127282 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.126847    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.127282 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.126954    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.127565 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.127537    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.128656 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.128102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.128656 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.128454    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.128819 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.128689    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-web-config\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.128819 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.128777    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.129234 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.129123    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.129234 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.129169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.129417 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.129396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.129829 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.129790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.129923 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.129842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.129923 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.129888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.130219 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.130174    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.130745 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.130724    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.131653 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.131630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-config-out\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.135664 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.135640    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t58n\" (UniqueName: \"kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-kube-api-access-7t58n\") pod \"prometheus-k8s-0\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:53.205291 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:53.205199    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:42:56.180090 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.179228    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"]
Aug 04 20:42:56.228321 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.228294    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 20:42:56.233817 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.233788    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 20:42:56.288955 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:42:56.288921    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efa6de3_c112_43e7_b24c_e383e7112126.slice/crio-06df131b102d9fffb86d740066fb15e3d033b1d6062aa06d6e83dc89d8bacec0 WatchSource:0}: Error finding container 06df131b102d9fffb86d740066fb15e3d033b1d6062aa06d6e83dc89d8bacec0: Status 404 returned error can't find the container with id 06df131b102d9fffb86d740066fb15e3d033b1d6062aa06d6e83dc89d8bacec0
Aug 04 20:42:56.419849 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.419810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-qjv89" event={"ID":"a86e1e49-9816-4c01-81a3-6e4850052b95","Type":"ContainerStarted","Data":"7b8f48627d8d46d96a75a77dfda624a85207b022fb35b3ded996028678edd846"}
Aug 04 20:42:56.420451 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.420431    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-qjv89"
Aug 04 20:42:56.422958 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.422444    2565 generic.go:356] "Generic (PLEG): container finished" podID="b59f4a25-b7f3-41c4-b839-2b3b2a906294" containerID="af17ef840348bf21c042e9c9d5b11b9201c5bd58f46ade8602e9d12dca8b59d7" exitCode=0
Aug 04 20:42:56.422958 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.422525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pp75k" event={"ID":"b59f4a25-b7f3-41c4-b839-2b3b2a906294","Type":"ContainerDied","Data":"af17ef840348bf21c042e9c9d5b11b9201c5bd58f46ade8602e9d12dca8b59d7"}
Aug 04 20:42:56.423874 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.423851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerStarted","Data":"463ebf582ccdf9f44d6a49c11babad087347f6c0a8a179237f7f56bcc03b0af3"}
Aug 04 20:42:56.425464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.425417    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-qjv89"
Aug 04 20:42:56.425959 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.425810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerStarted","Data":"06df131b102d9fffb86d740066fb15e3d033b1d6062aa06d6e83dc89d8bacec0"}
Aug 04 20:42:56.427524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.427480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db" event={"ID":"75b9271e-d59d-4700-a5c5-886da39c25d4","Type":"ContainerStarted","Data":"3606216cf6b89c711f977117002e0d8165315b9abdb5695c58ecdbbc60791e23"}
Aug 04 20:42:56.430427 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.430115    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8" event={"ID":"74c4817c-585c-48c7-82f9-95f4a0b1b958","Type":"ContainerStarted","Data":"8b081d934c29efe596d4e27ab219f1a9df52308260bab516b866d5c7272ff34f"}
Aug 04 20:42:56.430427 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.430154    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8" event={"ID":"74c4817c-585c-48c7-82f9-95f4a0b1b958","Type":"ContainerStarted","Data":"8f92e306be2152a6471d0e2e5f564b911b466d364aa44116ca5888eb8180e358"}
Aug 04 20:42:56.430427 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.430174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8" event={"ID":"74c4817c-585c-48c7-82f9-95f4a0b1b958","Type":"ContainerStarted","Data":"fabdb3fc868f54801ec57d360d4ab8f5b07feeb78e024f5f182a27a0e5ebca45"}
Aug 04 20:42:56.441723 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.441678    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-qjv89" podStartSLOduration=1.605055348 podStartE2EDuration="18.441666348s" totalImagesPullingTime="16.836611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:56.441402345 +0000 UTC m=+151.193508323" watchObservedRunningTime="2026-08-04 20:42:56.441666348 +0000 UTC m=+151.193772324"
Aug 04 20:42:56.526368 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:56.526231    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-dkgh8" podStartSLOduration=2.310526125 podStartE2EDuration="10.526209077s" totalImagesPullingTime="8.215682952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:56.525009254 +0000 UTC m=+151.277115268" watchObservedRunningTime="2026-08-04 20:42:56.526209077 +0000 UTC m=+151.278315055"
Aug 04 20:42:57.439182 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:57.439041    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pp75k" event={"ID":"b59f4a25-b7f3-41c4-b839-2b3b2a906294","Type":"ContainerStarted","Data":"e04fc135ddad0e4b4e0fd1a64de24800cb90af380a0334bacf8a774a23df9cf2"}
Aug 04 20:42:57.440249 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:57.439321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pp75k" event={"ID":"b59f4a25-b7f3-41c4-b839-2b3b2a906294","Type":"ContainerStarted","Data":"424bd4b8f11533861f47fea3d5483165c47e702c4701de8ad0cac5c568815ea7"}
Aug 04 20:42:57.462096 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:57.462034    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pp75k" podStartSLOduration=3.005736585 podStartE2EDuration="11.462014882s" totalImagesPullingTime="8.34688952s" totalInitContainerRuntime="109.388777ms" isStatefulPod=true podCreationTimestamp="2026-08-04 20:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:57.460840885 +0000 UTC m=+152.212946885" watchObservedRunningTime="2026-08-04 20:42:57.462014882 +0000 UTC m=+152.214120859"
Aug 04 20:42:59.458864 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:59.458822    2565 generic.go:356] "Generic (PLEG): container finished" podID="7efa6de3-c112-43e7-b24c-e383e7112126" containerID="4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1" exitCode=0
Aug 04 20:42:59.459337 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:59.458931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerDied","Data":"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"}
Aug 04 20:42:59.461273 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:59.460884    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db" event={"ID":"75b9271e-d59d-4700-a5c5-886da39c25d4","Type":"ContainerStarted","Data":"c4f15acd5c8423d90d8c06b335ba863eea07ec3745c559478ba587dac4d43e0c"}
Aug 04 20:42:59.461273 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:59.461136    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"
Aug 04 20:42:59.462798 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:59.462772    2565 generic.go:356] "Generic (PLEG): container finished" podID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerID="eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e" exitCode=0
Aug 04 20:42:59.462912 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:59.462828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerDied","Data":"eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e"}
Aug 04 20:42:59.468193 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:59.468168    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db"
Aug 04 20:42:59.536233 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:42:59.536180    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-jg6db" podStartSLOduration=6.09969106 podStartE2EDuration="8.536159403s" totalImagesPullingTime="2.436468343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:42:59.535628747 +0000 UTC m=+154.287734725" watchObservedRunningTime="2026-08-04 20:42:59.536159403 +0000 UTC m=+154.288265381"
Aug 04 20:43:00.686778 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:00.686149    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fbd494568-8vch9"]
Aug 04 20:43:00.686778 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:43:00.686562    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5fbd494568-8vch9" podUID="a8c1ad3c-f203-48b1-9368-866418f92fc5"
Aug 04 20:43:01.473251 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.473217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:43:01.501018 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.500984    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:43:01.523295 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.523265    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-certificates\") pod \"a8c1ad3c-f203-48b1-9368-866418f92fc5\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") "
Aug 04 20:43:01.523461 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.523304    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-bound-sa-token\") pod \"a8c1ad3c-f203-48b1-9368-866418f92fc5\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") "
Aug 04 20:43:01.523461 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.523360    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px5fh\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-kube-api-access-px5fh\") pod \"a8c1ad3c-f203-48b1-9368-866418f92fc5\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") "
Aug 04 20:43:01.523461 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.523415    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-trusted-ca\") pod \"a8c1ad3c-f203-48b1-9368-866418f92fc5\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") "
Aug 04 20:43:01.523461 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.523449    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8c1ad3c-f203-48b1-9368-866418f92fc5-ca-trust-extracted\") pod \"a8c1ad3c-f203-48b1-9368-866418f92fc5\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") "
Aug 04 20:43:01.523683 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.523481    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-image-registry-private-configuration\") pod \"a8c1ad3c-f203-48b1-9368-866418f92fc5\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") "
Aug 04 20:43:01.523683 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.523525    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-installation-pull-secrets\") pod \"a8c1ad3c-f203-48b1-9368-866418f92fc5\" (UID: \"a8c1ad3c-f203-48b1-9368-866418f92fc5\") "
Aug 04 20:43:01.524591 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.524521    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c1ad3c-f203-48b1-9368-866418f92fc5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a8c1ad3c-f203-48b1-9368-866418f92fc5" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 20:43:01.524885 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.524839    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a8c1ad3c-f203-48b1-9368-866418f92fc5" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 20:43:01.525676 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.525546    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a8c1ad3c-f203-48b1-9368-866418f92fc5" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 20:43:01.527458 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.527416    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a8c1ad3c-f203-48b1-9368-866418f92fc5" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:43:01.527611 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.527580    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-kube-api-access-px5fh" (OuterVolumeSpecName: "kube-api-access-px5fh") pod "a8c1ad3c-f203-48b1-9368-866418f92fc5" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5"). InnerVolumeSpecName "kube-api-access-px5fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 20:43:01.527837 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.527811    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a8c1ad3c-f203-48b1-9368-866418f92fc5" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 20:43:01.527926 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.527849    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a8c1ad3c-f203-48b1-9368-866418f92fc5" (UID: "a8c1ad3c-f203-48b1-9368-866418f92fc5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:43:01.625094 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.625015    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-trusted-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:43:01.625094 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.625052    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8c1ad3c-f203-48b1-9368-866418f92fc5-ca-trust-extracted\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:43:01.625094 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.625071    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-image-registry-private-configuration\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:43:01.625094 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.625086    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8c1ad3c-f203-48b1-9368-866418f92fc5-installation-pull-secrets\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:43:01.625094 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.625100    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-certificates\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:43:01.625637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.625113    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-bound-sa-token\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:43:01.625637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:01.625126    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px5fh\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-kube-api-access-px5fh\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:43:02.481364 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:02.481150    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbd494568-8vch9"
Aug 04 20:43:02.481364 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:02.481142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerStarted","Data":"db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab"}
Aug 04 20:43:02.519127 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:02.519089    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fbd494568-8vch9"]
Aug 04 20:43:02.527010 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:02.526952    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5fbd494568-8vch9"]
Aug 04 20:43:02.637316 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:02.637280    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8c1ad3c-f203-48b1-9368-866418f92fc5-registry-tls\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:43:03.591265 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:43:03.591233    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zr7n9" podUID="d6f2150f-4332-4f28-abe4-72de91210bfa"
Aug 04 20:43:03.760755 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:03.760723    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8c1ad3c-f203-48b1-9368-866418f92fc5" path="/var/lib/kubelet/pods/a8c1ad3c-f203-48b1-9368-866418f92fc5/volumes"
Aug 04 20:43:04.492088 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:04.492056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerStarted","Data":"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"}
Aug 04 20:43:04.492283 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:04.492185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerStarted","Data":"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"}
Aug 04 20:43:04.495814 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:04.495784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerStarted","Data":"29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09"}
Aug 04 20:43:04.495941 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:04.495825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:43:04.495941 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:04.495846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerStarted","Data":"643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423"}
Aug 04 20:43:04.495941 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:04.495882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerStarted","Data":"6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645"}
Aug 04 20:43:04.495941 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:04.495921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerStarted","Data":"1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327"}
Aug 04 20:43:06.508041 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:06.508009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerStarted","Data":"3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b"}
Aug 04 20:43:06.536809 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:06.536719    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=13.394748527 podStartE2EDuration="19.536700621s" totalImagesPullingTime="6.117084628s" totalInitContainerRuntime="24.867466ms" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:43:06.535113954 +0000 UTC m=+161.287219931" watchObservedRunningTime="2026-08-04 20:43:06.536700621 +0000 UTC m=+161.288806598"
Aug 04 20:43:07.516920 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:07.516882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerStarted","Data":"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"}
Aug 04 20:43:07.517481 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:07.516987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerStarted","Data":"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"}
Aug 04 20:43:07.517481 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:07.517021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerStarted","Data":"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"}
Aug 04 20:43:07.517481 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:07.517049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerStarted","Data":"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"}
Aug 04 20:43:07.554714 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:07.554657    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.406771831 podStartE2EDuration="15.554641726s" totalImagesPullingTime="9.125408459s" totalInitContainerRuntime="22.461436ms" isStatefulPod=false podCreationTimestamp="2026-08-04 20:42:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:43:07.553625085 +0000 UTC m=+162.305731062" watchObservedRunningTime="2026-08-04 20:43:07.554641726 +0000 UTC m=+162.306747702"
Aug 04 20:43:08.205509 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:08.205478    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:43:08.494953 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:08.494926    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:43:08.497310 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:08.497287    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f2150f-4332-4f28-abe4-72de91210bfa-metrics-tls\") pod \"dns-default-zr7n9\" (UID: \"d6f2150f-4332-4f28-abe4-72de91210bfa\") " pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:43:08.699219 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:08.699189    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kcgl6\""
Aug 04 20:43:08.707143 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:08.707125    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:43:08.867652 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:08.867627    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zr7n9"]
Aug 04 20:43:08.870123 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:43:08.870095    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f2150f_4332_4f28_abe4_72de91210bfa.slice/crio-e4921ef730cea31dc842b2ec5c730ee961d3fa920a3fa88c2de30f0a54afc940 WatchSource:0}: Error finding container e4921ef730cea31dc842b2ec5c730ee961d3fa920a3fa88c2de30f0a54afc940: Status 404 returned error can't find the container with id e4921ef730cea31dc842b2ec5c730ee961d3fa920a3fa88c2de30f0a54afc940
Aug 04 20:43:09.526637 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:09.526600    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zr7n9" event={"ID":"d6f2150f-4332-4f28-abe4-72de91210bfa","Type":"ContainerStarted","Data":"e4921ef730cea31dc842b2ec5c730ee961d3fa920a3fa88c2de30f0a54afc940"}
Aug 04 20:43:11.538258 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:11.538219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zr7n9" event={"ID":"d6f2150f-4332-4f28-abe4-72de91210bfa","Type":"ContainerStarted","Data":"2c1c4035dd5acf3f75408fe360e527dde3ee09a48b3848761bea2533852df8d7"}
Aug 04 20:43:11.538647 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:11.538323    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:43:11.538647 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:11.538357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zr7n9" event={"ID":"d6f2150f-4332-4f28-abe4-72de91210bfa","Type":"ContainerStarted","Data":"f654f5eadeab379d0a0718e81e3c0f7b0b1b054cc2aab176daa8b201e8eab3fb"}
Aug 04 20:43:11.560231 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:11.560189    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zr7n9" podStartSLOduration=129.218219275 podStartE2EDuration="2m11.560174931s" totalImagesPullingTime="2.341955656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:43:11.559511966 +0000 UTC m=+166.311617946" watchObservedRunningTime="2026-08-04 20:43:11.560174931 +0000 UTC m=+166.312280907"
Aug 04 20:43:21.544758 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:21.544680    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zr7n9"
Aug 04 20:43:27.599108 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:27.599070    2565 generic.go:356] "Generic (PLEG): container finished" podID="a0803eda-abb5-440a-8fcd-f4ffcb3257f6" containerID="d6e22f010f477559b79b9b7d9767f1e9e74d514f957560c9eec234203d25e154" exitCode=0
Aug 04 20:43:27.599593 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:27.599126    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv" event={"ID":"a0803eda-abb5-440a-8fcd-f4ffcb3257f6","Type":"ContainerDied","Data":"d6e22f010f477559b79b9b7d9767f1e9e74d514f957560c9eec234203d25e154"}
Aug 04 20:43:27.599593 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:27.599504    2565 scope.go:117] "RemoveContainer" containerID="d6e22f010f477559b79b9b7d9767f1e9e74d514f957560c9eec234203d25e154"
Aug 04 20:43:28.604838 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:28.604803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-2qttv" event={"ID":"a0803eda-abb5-440a-8fcd-f4ffcb3257f6","Type":"ContainerStarted","Data":"d62055b37652cf77fd3b92abb2d1c02ff8bb2c8137a42567a38cc8181cc2f463"}
Aug 04 20:43:32.623247 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:32.623214    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb97b82b-f635-4e6d-8c35-354c0e67ef49" containerID="9b38ed6c5a7f04d4fe423d757d9092e387a22baeae5eb85eb4c64a73671c3b8e" exitCode=0
Aug 04 20:43:32.623621 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:32.623258    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z" event={"ID":"cb97b82b-f635-4e6d-8c35-354c0e67ef49","Type":"ContainerDied","Data":"9b38ed6c5a7f04d4fe423d757d9092e387a22baeae5eb85eb4c64a73671c3b8e"}
Aug 04 20:43:32.623621 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:32.623590    2565 scope.go:117] "RemoveContainer" containerID="9b38ed6c5a7f04d4fe423d757d9092e387a22baeae5eb85eb4c64a73671c3b8e"
Aug 04 20:43:33.628929 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:33.628888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-fdk5z" event={"ID":"cb97b82b-f635-4e6d-8c35-354c0e67ef49","Type":"ContainerStarted","Data":"867fb3d549860e1f698eccf3b59fa04d71d0c57b528923bba3fa349d068d6dfd"}
Aug 04 20:43:53.206039 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:53.206002    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:43:53.224614 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:53.224592    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:43:53.721764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:43:53.721734    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:06.959251 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:06.959220    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 20:44:06.959699 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:06.959514    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="alertmanager" containerID="cri-o://db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab" gracePeriod=120
Aug 04 20:44:06.959699 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:06.959561    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy-metric" containerID="cri-o://29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09" gracePeriod=120
Aug 04 20:44:06.959699 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:06.959613    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy-web" containerID="cri-o://6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645" gracePeriod=120
Aug 04 20:44:06.959699 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:06.959644    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="config-reloader" containerID="cri-o://1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327" gracePeriod=120
Aug 04 20:44:06.959960 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:06.959718    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="prom-label-proxy" containerID="cri-o://3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b" gracePeriod=120
Aug 04 20:44:06.959960 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:06.959651    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy" containerID="cri-o://643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423" gracePeriod=120
Aug 04 20:44:07.772571 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:07.772539    2565 generic.go:356] "Generic (PLEG): container finished" podID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerID="3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b" exitCode=0
Aug 04 20:44:07.772571 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:07.772563    2565 generic.go:356] "Generic (PLEG): container finished" podID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerID="643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423" exitCode=0
Aug 04 20:44:07.772571 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:07.772570    2565 generic.go:356] "Generic (PLEG): container finished" podID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerID="1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327" exitCode=0
Aug 04 20:44:07.772571 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:07.772575    2565 generic.go:356] "Generic (PLEG): container finished" podID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerID="db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab" exitCode=0
Aug 04 20:44:07.772846 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:07.772606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerDied","Data":"3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b"}
Aug 04 20:44:07.772846 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:07.772670    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerDied","Data":"643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423"}
Aug 04 20:44:07.772846 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:07.772696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerDied","Data":"1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327"}
Aug 04 20:44:07.772846 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:07.772721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerDied","Data":"db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab"}
Aug 04 20:44:08.198969 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.198946    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.292916 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.292838    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-web\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.292916 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.292890    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-metric\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293121 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.292918    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-tls-assets\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293121 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.292965    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293121 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.292993    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-main-db\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293121 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293017    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-cluster-tls-config\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293121 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293043    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-main-tls\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293121 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293068    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-out\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293121 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293119    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-web-config\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293511 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293157    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b74kv\" (UniqueName: \"kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-kube-api-access-b74kv\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293511 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293197    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-metrics-client-ca\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293511 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293237    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-volume\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293511 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293276    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-trusted-ca-bundle\") pod \"da807b0b-ed13-4af1-967d-c79cd5d5b058\" (UID: \"da807b0b-ed13-4af1-967d-c79cd5d5b058\") "
Aug 04 20:44:08.293511 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293322    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 20:44:08.293759 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293617    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-main-db\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.293759 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293695    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 20:44:08.294017 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.293993    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 20:44:08.296865 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.296719    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:08.296865 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.296797    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-kube-api-access-b74kv" (OuterVolumeSpecName: "kube-api-access-b74kv") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "kube-api-access-b74kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 20:44:08.296865 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.296808    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-volume" (OuterVolumeSpecName: "config-volume") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:08.296865 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.296800    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 20:44:08.296865 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.296828    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:08.297163 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.296948    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:08.297321 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.297301    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-out" (OuterVolumeSpecName: "config-out") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 20:44:08.298446 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.298425    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:08.299975 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.299953    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:08.309640 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.309621    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-web-config" (OuterVolumeSpecName: "web-config") pod "da807b0b-ed13-4af1-967d-c79cd5d5b058" (UID: "da807b0b-ed13-4af1-967d-c79cd5d5b058"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:08.394748 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394716    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.394748 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394739    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-tls-assets\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.394748 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394751    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.394965 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394761    2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-cluster-tls-config\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.394965 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394771    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-main-tls\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.394965 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394779    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-out\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.394965 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394787    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-web-config\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.394965 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394795    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b74kv\" (UniqueName: \"kubernetes.io/projected/da807b0b-ed13-4af1-967d-c79cd5d5b058-kube-api-access-b74kv\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.394965 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394804    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-metrics-client-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.394965 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394813    2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.394965 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394821    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da807b0b-ed13-4af1-967d-c79cd5d5b058-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.394965 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.394829    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da807b0b-ed13-4af1-967d-c79cd5d5b058-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:08.779448 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.779413    2565 generic.go:356] "Generic (PLEG): container finished" podID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerID="29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09" exitCode=0
Aug 04 20:44:08.779448 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.779442    2565 generic.go:356] "Generic (PLEG): container finished" podID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerID="6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645" exitCode=0
Aug 04 20:44:08.779651 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.779505    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.779651 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.779496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerDied","Data":"29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09"}
Aug 04 20:44:08.779651 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.779637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerDied","Data":"6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645"}
Aug 04 20:44:08.779853 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.779665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da807b0b-ed13-4af1-967d-c79cd5d5b058","Type":"ContainerDied","Data":"463ebf582ccdf9f44d6a49c11babad087347f6c0a8a179237f7f56bcc03b0af3"}
Aug 04 20:44:08.779853 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.779703    2565 scope.go:117] "RemoveContainer" containerID="3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b"
Aug 04 20:44:08.787205 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.787187    2565 scope.go:117] "RemoveContainer" containerID="29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09"
Aug 04 20:44:08.794657 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.794635    2565 scope.go:117] "RemoveContainer" containerID="643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423"
Aug 04 20:44:08.801197 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.801153    2565 scope.go:117] "RemoveContainer" containerID="6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645"
Aug 04 20:44:08.804201 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.804181    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 20:44:08.809224 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.809199    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 20:44:08.809792 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.809771    2565 scope.go:117] "RemoveContainer" containerID="1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327"
Aug 04 20:44:08.816384 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.816366    2565 scope.go:117] "RemoveContainer" containerID="db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab"
Aug 04 20:44:08.823269 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.823252    2565 scope.go:117] "RemoveContainer" containerID="eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e"
Aug 04 20:44:08.830019 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.830004    2565 scope.go:117] "RemoveContainer" containerID="3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b"
Aug 04 20:44:08.830264 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:08.830238    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b\": container with ID starting with 3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b not found: ID does not exist" containerID="3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b"
Aug 04 20:44:08.830314 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.830274    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b"} err="failed to get container status \"3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b\": rpc error: code = NotFound desc = could not find container \"3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b\": container with ID starting with 3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b not found: ID does not exist"
Aug 04 20:44:08.830314 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.830306    2565 scope.go:117] "RemoveContainer" containerID="29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09"
Aug 04 20:44:08.830571 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:08.830554    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09\": container with ID starting with 29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09 not found: ID does not exist" containerID="29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09"
Aug 04 20:44:08.830621 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.830574    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09"} err="failed to get container status \"29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09\": rpc error: code = NotFound desc = could not find container \"29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09\": container with ID starting with 29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09 not found: ID does not exist"
Aug 04 20:44:08.830621 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.830591    2565 scope.go:117] "RemoveContainer" containerID="643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423"
Aug 04 20:44:08.830804 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:08.830788    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423\": container with ID starting with 643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423 not found: ID does not exist" containerID="643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423"
Aug 04 20:44:08.830846 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.830810    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423"} err="failed to get container status \"643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423\": rpc error: code = NotFound desc = could not find container \"643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423\": container with ID starting with 643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423 not found: ID does not exist"
Aug 04 20:44:08.830846 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.830825    2565 scope.go:117] "RemoveContainer" containerID="6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645"
Aug 04 20:44:08.831030 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:08.831013    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645\": container with ID starting with 6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645 not found: ID does not exist" containerID="6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645"
Aug 04 20:44:08.831069 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.831033    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645"} err="failed to get container status \"6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645\": rpc error: code = NotFound desc = could not find container \"6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645\": container with ID starting with 6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645 not found: ID does not exist"
Aug 04 20:44:08.831069 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.831045    2565 scope.go:117] "RemoveContainer" containerID="1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327"
Aug 04 20:44:08.831254 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:08.831233    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327\": container with ID starting with 1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327 not found: ID does not exist" containerID="1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327"
Aug 04 20:44:08.831303 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.831250    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327"} err="failed to get container status \"1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327\": rpc error: code = NotFound desc = could not find container \"1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327\": container with ID starting with 1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327 not found: ID does not exist"
Aug 04 20:44:08.831303 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.831262    2565 scope.go:117] "RemoveContainer" containerID="db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab"
Aug 04 20:44:08.831522 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:08.831503    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab\": container with ID starting with db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab not found: ID does not exist" containerID="db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab"
Aug 04 20:44:08.831564 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.831529    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab"} err="failed to get container status \"db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab\": rpc error: code = NotFound desc = could not find container \"db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab\": container with ID starting with db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab not found: ID does not exist"
Aug 04 20:44:08.831564 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.831543    2565 scope.go:117] "RemoveContainer" containerID="eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e"
Aug 04 20:44:08.831788 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:08.831770    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e\": container with ID starting with eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e not found: ID does not exist" containerID="eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e"
Aug 04 20:44:08.831831 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.831793    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e"} err="failed to get container status \"eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e\": rpc error: code = NotFound desc = could not find container \"eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e\": container with ID starting with eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e not found: ID does not exist"
Aug 04 20:44:08.831831 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.831808    2565 scope.go:117] "RemoveContainer" containerID="3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b"
Aug 04 20:44:08.832040 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.832020    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b"} err="failed to get container status \"3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b\": rpc error: code = NotFound desc = could not find container \"3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b\": container with ID starting with 3b0f1920babc23ec1a646fc70aecb38f263f70bc76b9dbc7c715bccedb43a18b not found: ID does not exist"
Aug 04 20:44:08.832092 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.832041    2565 scope.go:117] "RemoveContainer" containerID="29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09"
Aug 04 20:44:08.832464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.832257    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09"} err="failed to get container status \"29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09\": rpc error: code = NotFound desc = could not find container \"29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09\": container with ID starting with 29ab3071d785dde976cceb4eb085d8ac219836a4e5401461691fdaca6d985e09 not found: ID does not exist"
Aug 04 20:44:08.832464 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.832307    2565 scope.go:117] "RemoveContainer" containerID="643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423"
Aug 04 20:44:08.832668 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.832607    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423"} err="failed to get container status \"643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423\": rpc error: code = NotFound desc = could not find container \"643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423\": container with ID starting with 643214b3fd179662b9525fba58e9881646da778034506739493bde183b8a3423 not found: ID does not exist"
Aug 04 20:44:08.832668 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.832642    2565 scope.go:117] "RemoveContainer" containerID="6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645"
Aug 04 20:44:08.833369 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.832989    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645"} err="failed to get container status \"6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645\": rpc error: code = NotFound desc = could not find container \"6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645\": container with ID starting with 6258b82f5614db4ba34761413247bd786a6dd2a8bd0620a1f17a67bf1f5a9645 not found: ID does not exist"
Aug 04 20:44:08.833369 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.833015    2565 scope.go:117] "RemoveContainer" containerID="1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327"
Aug 04 20:44:08.833369 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.833311    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327"} err="failed to get container status \"1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327\": rpc error: code = NotFound desc = could not find container \"1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327\": container with ID starting with 1ca3479337969333fe9b5a0f6859f20d93738031054a205ad6e09027719f3327 not found: ID does not exist"
Aug 04 20:44:08.833369 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.833330    2565 scope.go:117] "RemoveContainer" containerID="db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab"
Aug 04 20:44:08.833641 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.833619    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab"} err="failed to get container status \"db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab\": rpc error: code = NotFound desc = could not find container \"db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab\": container with ID starting with db185f7b4ecbc427b200ff011334450f38d10826cb757ad7915e2793900f6cab not found: ID does not exist"
Aug 04 20:44:08.833696 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.833643    2565 scope.go:117] "RemoveContainer" containerID="eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e"
Aug 04 20:44:08.833939 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.833919    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e"} err="failed to get container status \"eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e\": rpc error: code = NotFound desc = could not find container \"eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e\": container with ID starting with eaa0acd00daae66f05a5105410964da6a2411f67ff76272e6a9d2832d3bdb33e not found: ID does not exist"
Aug 04 20:44:08.834570 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834551    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 20:44:08.834835 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834823    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy-metric"
Aug 04 20:44:08.834879 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834836    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy-metric"
Aug 04 20:44:08.834879 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834846    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="prom-label-proxy"
Aug 04 20:44:08.834879 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834852    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="prom-label-proxy"
Aug 04 20:44:08.834879 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834859    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="alertmanager"
Aug 04 20:44:08.834879 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834865    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="alertmanager"
Aug 04 20:44:08.834879 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834877    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="config-reloader"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834882    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="config-reloader"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834894    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy-web"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834899    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy-web"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834906    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="init-config-reloader"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834911    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="init-config-reloader"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834918    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834923    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834969    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy-metric"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834977    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="config-reloader"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834984    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834990    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="prom-label-proxy"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.834998    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="alertmanager"
Aug 04 20:44:08.835047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.835004    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" containerName="kube-rbac-proxy-web"
Aug 04 20:44:08.839909 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.839893    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.842537 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.842514    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Aug 04 20:44:08.842613 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.842591    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Aug 04 20:44:08.842707 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.842693    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Aug 04 20:44:08.842763 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.842709    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Aug 04 20:44:08.842763 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.842712    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Aug 04 20:44:08.842835 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.842815    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-87kjp\""
Aug 04 20:44:08.843077 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.843061    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Aug 04 20:44:08.843150 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.843083    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Aug 04 20:44:08.843239 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.843223    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Aug 04 20:44:08.848467 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.848446    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Aug 04 20:44:08.853985 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.853965    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 20:44:08.899243 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899217    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899325 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899325 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899281    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899420 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qxg\" (UniqueName: \"kubernetes.io/projected/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-kube-api-access-k2qxg\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899420 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899420 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899399    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899507 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899420    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-config-volume\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899507 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-config-out\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899507 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899507 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899624 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-web-config\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899624 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899551    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:08.899624 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:08.899566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000335 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-web-config\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000479 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000479 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000479 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000609 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000609 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000609 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qxg\" (UniqueName: \"kubernetes.io/projected/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-kube-api-access-k2qxg\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000736 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000605    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000736 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000736 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-config-volume\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000736 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-config-out\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000736 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.000970 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.000773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.001411 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.001374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.002080 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.002001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.003514 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.003490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.003612 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.003493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.003612 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.003565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.003729 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.003634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.003729 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.003638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-config-volume\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.003897 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.003872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-config-out\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.004487 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.004462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-web-config\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.004551 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.004515    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.004597 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.004554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.005512 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.005495    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.009097 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.009073    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qxg\" (UniqueName: \"kubernetes.io/projected/e1fdccb7-12a2-4f1e-98a0-1b688b22999f-kube-api-access-k2qxg\") pod \"alertmanager-main-0\" (UID: \"e1fdccb7-12a2-4f1e-98a0-1b688b22999f\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.149853 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.149823    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 20:44:09.288094 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.288053    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 20:44:09.290434 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:44:09.290380    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fdccb7_12a2_4f1e_98a0_1b688b22999f.slice/crio-074cd64fee96d26eb54f3e0e5a1dc03d24c090e64ec30b7ced6e579434ed52eb WatchSource:0}: Error finding container 074cd64fee96d26eb54f3e0e5a1dc03d24c090e64ec30b7ced6e579434ed52eb: Status 404 returned error can't find the container with id 074cd64fee96d26eb54f3e0e5a1dc03d24c090e64ec30b7ced6e579434ed52eb
Aug 04 20:44:09.759407 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.759373    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da807b0b-ed13-4af1-967d-c79cd5d5b058" path="/var/lib/kubelet/pods/da807b0b-ed13-4af1-967d-c79cd5d5b058/volumes"
Aug 04 20:44:09.784911 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.784882    2565 generic.go:356] "Generic (PLEG): container finished" podID="e1fdccb7-12a2-4f1e-98a0-1b688b22999f" containerID="2216c87c53341333dd761c99d1aff7569683a71e0a7e47ca9ae469f9b992a2cd" exitCode=0
Aug 04 20:44:09.785056 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.784967    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1fdccb7-12a2-4f1e-98a0-1b688b22999f","Type":"ContainerDied","Data":"2216c87c53341333dd761c99d1aff7569683a71e0a7e47ca9ae469f9b992a2cd"}
Aug 04 20:44:09.785056 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:09.785035    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1fdccb7-12a2-4f1e-98a0-1b688b22999f","Type":"ContainerStarted","Data":"074cd64fee96d26eb54f3e0e5a1dc03d24c090e64ec30b7ced6e579434ed52eb"}
Aug 04 20:44:10.792830 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.792786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1fdccb7-12a2-4f1e-98a0-1b688b22999f","Type":"ContainerStarted","Data":"f34c941fd04f42f4561021450fa418570e066a081a11d5cc988389d5e9edf8f2"}
Aug 04 20:44:10.793238 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.792880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1fdccb7-12a2-4f1e-98a0-1b688b22999f","Type":"ContainerStarted","Data":"eef7fcb08d6173886aefd75de9e87c5ee5d1a779cae455b931e4b4581c516cfc"}
Aug 04 20:44:10.793238 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.792944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1fdccb7-12a2-4f1e-98a0-1b688b22999f","Type":"ContainerStarted","Data":"26f3716787554651106f14d4d4767c1d588e67cf1228e27bc9b074c2d18494cb"}
Aug 04 20:44:10.793238 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.792985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1fdccb7-12a2-4f1e-98a0-1b688b22999f","Type":"ContainerStarted","Data":"a1c2206c19e591fe7e381383d700b62ae7297620f8de82028fd93bfa8ae64b17"}
Aug 04 20:44:10.793238 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.793029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1fdccb7-12a2-4f1e-98a0-1b688b22999f","Type":"ContainerStarted","Data":"2a45cdd2c811c14c0c9015b935dc4089f0fc0be03deb3efd2599e35219105c15"}
Aug 04 20:44:10.793238 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.793070    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1fdccb7-12a2-4f1e-98a0-1b688b22999f","Type":"ContainerStarted","Data":"42e5d29e280216fda678cb915bdf83cfaff47af82f2a46492ed092428cb52c43"}
Aug 04 20:44:10.823617 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.823559    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.808908794 podStartE2EDuration="2.823540112s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.631318ms" isStatefulPod=false podCreationTimestamp="2026-08-04 20:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:44:10.820535535 +0000 UTC m=+225.572641525" watchObservedRunningTime="2026-08-04 20:44:10.823540112 +0000 UTC m=+225.575646090"
Aug 04 20:44:10.972679 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.972644    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-649c45fdd-rjc6n"]
Aug 04 20:44:10.976322 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.976301    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:10.979013 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.978980    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Aug 04 20:44:10.979143 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.979049    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Aug 04 20:44:10.979143 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.979057    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Aug 04 20:44:10.979143 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.979057    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Aug 04 20:44:10.979143 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.979094    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d7ncg\""
Aug 04 20:44:10.979316 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.978989    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Aug 04 20:44:10.985421 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.985399    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Aug 04 20:44:10.993620 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:10.993597    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-649c45fdd-rjc6n"]
Aug 04 20:44:11.016826 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.016801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.016928 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.016828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-federate-client-tls\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.016928 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.016853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c91f55b7-fdea-4528-bcc7-3a283561bab7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.016928 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.016885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-telemeter-client-tls\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.016928 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.016911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c91f55b7-fdea-4528-bcc7-3a283561bab7-serving-certs-ca-bundle\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.016928 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.016925    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c91f55b7-fdea-4528-bcc7-3a283561bab7-metrics-client-ca\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.017120 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.016944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghk8\" (UniqueName: \"kubernetes.io/projected/c91f55b7-fdea-4528-bcc7-3a283561bab7-kube-api-access-bghk8\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.017158 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.017109    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-secret-telemeter-client\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.117773 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.117680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.117773 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.117717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-federate-client-tls\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.117773 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.117737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c91f55b7-fdea-4528-bcc7-3a283561bab7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.117773 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.117761    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-telemeter-client-tls\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.118079 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.117792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c91f55b7-fdea-4528-bcc7-3a283561bab7-serving-certs-ca-bundle\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.118079 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.117820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c91f55b7-fdea-4528-bcc7-3a283561bab7-metrics-client-ca\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.118079 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.117844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghk8\" (UniqueName: \"kubernetes.io/projected/c91f55b7-fdea-4528-bcc7-3a283561bab7-kube-api-access-bghk8\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.118079 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.117959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-secret-telemeter-client\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.118740 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.118708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c91f55b7-fdea-4528-bcc7-3a283561bab7-metrics-client-ca\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.118740 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.118708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c91f55b7-fdea-4528-bcc7-3a283561bab7-serving-certs-ca-bundle\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.118933 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.118869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c91f55b7-fdea-4528-bcc7-3a283561bab7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.120832 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.120806    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.120942 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.120834    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-federate-client-tls\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.120942 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.120914    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-secret-telemeter-client\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.120942 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.120924    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c91f55b7-fdea-4528-bcc7-3a283561bab7-telemeter-client-tls\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.127571 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.127549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghk8\" (UniqueName: \"kubernetes.io/projected/c91f55b7-fdea-4528-bcc7-3a283561bab7-kube-api-access-bghk8\") pod \"telemeter-client-649c45fdd-rjc6n\" (UID: \"c91f55b7-fdea-4528-bcc7-3a283561bab7\") " pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.257056 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.257016    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 20:44:11.257357 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.257294    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="prometheus" containerID="cri-o://0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447" gracePeriod=600
Aug 04 20:44:11.257443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.257357    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="config-reloader" containerID="cri-o://4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157" gracePeriod=600
Aug 04 20:44:11.257443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.257372    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy" containerID="cri-o://00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64" gracePeriod=600
Aug 04 20:44:11.257443 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.257427    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9" gracePeriod=600
Aug 04 20:44:11.257649 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.257381    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="thanos-sidecar" containerID="cri-o://559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32" gracePeriod=600
Aug 04 20:44:11.257649 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.257415    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy-web" containerID="cri-o://20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee" gracePeriod=600
Aug 04 20:44:11.287418 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.287391    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n"
Aug 04 20:44:11.435062 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.435032    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-649c45fdd-rjc6n"]
Aug 04 20:44:11.439442 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:44:11.439416    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91f55b7_fdea_4528_bcc7_3a283561bab7.slice/crio-b2beca28e9819453dff4535e01f4949968b0e149922eeb9e33dfd7b0784496d6 WatchSource:0}: Error finding container b2beca28e9819453dff4535e01f4949968b0e149922eeb9e33dfd7b0784496d6: Status 404 returned error can't find the container with id b2beca28e9819453dff4535e01f4949968b0e149922eeb9e33dfd7b0784496d6
Aug 04 20:44:11.500145 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.500120    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.622625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622535    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-thanos-prometheus-http-client-file\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.622625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622610    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-web-config\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.622798 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622637    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t58n\" (UniqueName: \"kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-kube-api-access-7t58n\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.622798 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622670    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.622798 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622708    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-0\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.622798 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622742    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-metrics-client-certs\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.622798 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622767    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-tls\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623040 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622803    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-2\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623040 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622839    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-trusted-ca-bundle\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623040 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622864    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-serving-certs-ca-bundle\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623040 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622891    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-1\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623040 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622931    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-db\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623040 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622956    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-config-out\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623040 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.622977    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-kubelet-serving-ca-bundle\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623040 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.623013    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-config\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623479 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.623055    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-metrics-client-ca\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623479 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.623088    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-tls-assets\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623479 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.623128    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-kube-rbac-proxy\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623479 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.623179    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623479 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.623212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-grpc-tls\") pod \"7efa6de3-c112-43e7-b24c-e383e7112126\" (UID: \"7efa6de3-c112-43e7-b24c-e383e7112126\") "
Aug 04 20:44:11.623479 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.623290    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 20:44:11.623803 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.623571    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.623803 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.623692    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 20:44:11.625097 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.624180    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 20:44:11.625097 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.624891    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 20:44:11.625097 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.624987    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 20:44:11.625097 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.625077    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 20:44:11.625747 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.625553    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 20:44:11.625747 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.625642    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-kube-api-access-7t58n" (OuterVolumeSpecName: "kube-api-access-7t58n") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "kube-api-access-7t58n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 20:44:11.625747 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.625712    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:11.625986 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.625855    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 20:44:11.626172 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.626130    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:11.626861 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.626825    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:11.627302 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.627263    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:11.627581 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.627549    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:11.627786 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.627766    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-config-out" (OuterVolumeSpecName: "config-out") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 20:44:11.627786 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.627773    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-config" (OuterVolumeSpecName: "config") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:11.627929 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.627908    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 20:44:11.628129 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.628107    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:11.628590 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.628572    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:11.636853 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.636835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-web-config" (OuterVolumeSpecName: "web-config") pod "7efa6de3-c112-43e7-b24c-e383e7112126" (UID: "7efa6de3-c112-43e7-b24c-e383e7112126"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 20:44:11.724496 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724463    2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-metrics-client-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724496 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724489    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724496 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724499    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724509    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724518    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724526    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-db\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724535    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7efa6de3-c112-43e7-b24c-e383e7112126-config-out\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724544    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724552    2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-config\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724561    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-configmap-metrics-client-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724568    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-tls-assets\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724577    2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-kube-rbac-proxy\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724585    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724593    2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-grpc-tls\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724602    2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724610    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-web-config\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724618    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t58n\" (UniqueName: \"kubernetes.io/projected/7efa6de3-c112-43e7-b24c-e383e7112126-kube-api-access-7t58n\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724626    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7efa6de3-c112-43e7-b24c-e383e7112126-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.724684 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.724634    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7efa6de3-c112-43e7-b24c-e383e7112126-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:44:11.802878 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.802840    2565 generic.go:356] "Generic (PLEG): container finished" podID="7efa6de3-c112-43e7-b24c-e383e7112126" containerID="8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9" exitCode=0
Aug 04 20:44:11.802878 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.802866    2565 generic.go:356] "Generic (PLEG): container finished" podID="7efa6de3-c112-43e7-b24c-e383e7112126" containerID="00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64" exitCode=0
Aug 04 20:44:11.802878 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.802872    2565 generic.go:356] "Generic (PLEG): container finished" podID="7efa6de3-c112-43e7-b24c-e383e7112126" containerID="20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee" exitCode=0
Aug 04 20:44:11.802878 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.802882    2565 generic.go:356] "Generic (PLEG): container finished" podID="7efa6de3-c112-43e7-b24c-e383e7112126" containerID="559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32" exitCode=0
Aug 04 20:44:11.802878 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.802887    2565 generic.go:356] "Generic (PLEG): container finished" podID="7efa6de3-c112-43e7-b24c-e383e7112126" containerID="4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157" exitCode=0
Aug 04 20:44:11.803471 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.802892    2565 generic.go:356] "Generic (PLEG): container finished" podID="7efa6de3-c112-43e7-b24c-e383e7112126" containerID="0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447" exitCode=0
Aug 04 20:44:11.803471 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.802921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerDied","Data":"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"}
Aug 04 20:44:11.803471 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.802947    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.803471 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.802993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerDied","Data":"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"}
Aug 04 20:44:11.803471 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.803016    2565 scope.go:117] "RemoveContainer" containerID="8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"
Aug 04 20:44:11.803471 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.803041    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerDied","Data":"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"}
Aug 04 20:44:11.803471 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.803083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerDied","Data":"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"}
Aug 04 20:44:11.803471 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.803120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerDied","Data":"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"}
Aug 04 20:44:11.803471 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.803160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerDied","Data":"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"}
Aug 04 20:44:11.803471 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.803202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7efa6de3-c112-43e7-b24c-e383e7112126","Type":"ContainerDied","Data":"06df131b102d9fffb86d740066fb15e3d033b1d6062aa06d6e83dc89d8bacec0"}
Aug 04 20:44:11.804518 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.804495    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n" event={"ID":"c91f55b7-fdea-4528-bcc7-3a283561bab7","Type":"ContainerStarted","Data":"b2beca28e9819453dff4535e01f4949968b0e149922eeb9e33dfd7b0784496d6"}
Aug 04 20:44:11.811102 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.811085    2565 scope.go:117] "RemoveContainer" containerID="00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"
Aug 04 20:44:11.818376 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.818352    2565 scope.go:117] "RemoveContainer" containerID="20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"
Aug 04 20:44:11.825101 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.825083    2565 scope.go:117] "RemoveContainer" containerID="559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"
Aug 04 20:44:11.831558 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.827765    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 20:44:11.833063 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.833038    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 20:44:11.837685 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.837661    2565 scope.go:117] "RemoveContainer" containerID="4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"
Aug 04 20:44:11.844421 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.844397    2565 scope.go:117] "RemoveContainer" containerID="0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"
Aug 04 20:44:11.851391 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.851377    2565 scope.go:117] "RemoveContainer" containerID="4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"
Aug 04 20:44:11.857937 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.857915    2565 scope.go:117] "RemoveContainer" containerID="8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"
Aug 04 20:44:11.858240 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:11.858215    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": container with ID starting with 8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9 not found: ID does not exist" containerID="8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"
Aug 04 20:44:11.858338 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.858252    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"} err="failed to get container status \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": rpc error: code = NotFound desc = could not find container \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": container with ID starting with 8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9 not found: ID does not exist"
Aug 04 20:44:11.858338 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.858284    2565 scope.go:117] "RemoveContainer" containerID="00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"
Aug 04 20:44:11.858612 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:11.858588    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": container with ID starting with 00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64 not found: ID does not exist" containerID="00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"
Aug 04 20:44:11.858693 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.858619    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"} err="failed to get container status \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": rpc error: code = NotFound desc = could not find container \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": container with ID starting with 00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64 not found: ID does not exist"
Aug 04 20:44:11.858693 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.858636    2565 scope.go:117] "RemoveContainer" containerID="20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"
Aug 04 20:44:11.858989 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:11.858969    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": container with ID starting with 20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee not found: ID does not exist" containerID="20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"
Aug 04 20:44:11.859059 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.858994    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"} err="failed to get container status \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": rpc error: code = NotFound desc = could not find container \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": container with ID starting with 20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee not found: ID does not exist"
Aug 04 20:44:11.859059 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.859011    2565 scope.go:117] "RemoveContainer" containerID="559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"
Aug 04 20:44:11.859289 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:11.859261    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": container with ID starting with 559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32 not found: ID does not exist" containerID="559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"
Aug 04 20:44:11.859386 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.859294    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"} err="failed to get container status \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": rpc error: code = NotFound desc = could not find container \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": container with ID starting with 559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32 not found: ID does not exist"
Aug 04 20:44:11.859386 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.859309    2565 scope.go:117] "RemoveContainer" containerID="4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"
Aug 04 20:44:11.859573 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:11.859557    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": container with ID starting with 4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157 not found: ID does not exist" containerID="4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"
Aug 04 20:44:11.859627 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.859578    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"} err="failed to get container status \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": rpc error: code = NotFound desc = could not find container \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": container with ID starting with 4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157 not found: ID does not exist"
Aug 04 20:44:11.859627 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.859592    2565 scope.go:117] "RemoveContainer" containerID="0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"
Aug 04 20:44:11.859860 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:11.859843    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": container with ID starting with 0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447 not found: ID does not exist" containerID="0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"
Aug 04 20:44:11.859918 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.859865    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"} err="failed to get container status \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": rpc error: code = NotFound desc = could not find container \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": container with ID starting with 0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447 not found: ID does not exist"
Aug 04 20:44:11.859918 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.859879    2565 scope.go:117] "RemoveContainer" containerID="4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"
Aug 04 20:44:11.860156 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:44:11.860140    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": container with ID starting with 4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1 not found: ID does not exist" containerID="4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"
Aug 04 20:44:11.860203 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860161    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"} err="failed to get container status \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": rpc error: code = NotFound desc = could not find container \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": container with ID starting with 4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1 not found: ID does not exist"
Aug 04 20:44:11.860203 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860172    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 20:44:11.860287 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860175    2565 scope.go:117] "RemoveContainer" containerID="8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"
Aug 04 20:44:11.860498 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860482    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"} err="failed to get container status \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": rpc error: code = NotFound desc = could not find container \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": container with ID starting with 8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9 not found: ID does not exist"
Aug 04 20:44:11.860554 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860498    2565 scope.go:117] "RemoveContainer" containerID="00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"
Aug 04 20:44:11.860585 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860547    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy-thanos"
Aug 04 20:44:11.860585 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860561    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy-thanos"
Aug 04 20:44:11.860585 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860570    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy"
Aug 04 20:44:11.860585 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860575    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy"
Aug 04 20:44:11.860585 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860584    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="thanos-sidecar"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860589    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="thanos-sidecar"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860596    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy-web"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860601    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy-web"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860609    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="init-config-reloader"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860615    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="init-config-reloader"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860628    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="config-reloader"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860634    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="config-reloader"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860641    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="prometheus"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860646    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="prometheus"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860690    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="thanos-sidecar"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860701    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="prometheus"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860708    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="config-reloader"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860714    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy-web"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860720    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy-thanos"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860725    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" containerName="kube-rbac-proxy"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860729    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"} err="failed to get container status \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": rpc error: code = NotFound desc = could not find container \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": container with ID starting with 00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64 not found: ID does not exist"
Aug 04 20:44:11.860764 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860746    2565 scope.go:117] "RemoveContainer" containerID="20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"
Aug 04 20:44:11.861355 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860953    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"} err="failed to get container status \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": rpc error: code = NotFound desc = could not find container \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": container with ID starting with 20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee not found: ID does not exist"
Aug 04 20:44:11.861355 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.860977    2565 scope.go:117] "RemoveContainer" containerID="559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"
Aug 04 20:44:11.861355 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.861194    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"} err="failed to get container status \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": rpc error: code = NotFound desc = could not find container \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": container with ID starting with 559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32 not found: ID does not exist"
Aug 04 20:44:11.861355 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.861211    2565 scope.go:117] "RemoveContainer" containerID="4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"
Aug 04 20:44:11.861491 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.861453    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"} err="failed to get container status \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": rpc error: code = NotFound desc = could not find container \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": container with ID starting with 4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157 not found: ID does not exist"
Aug 04 20:44:11.861491 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.861474    2565 scope.go:117] "RemoveContainer" containerID="0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"
Aug 04 20:44:11.861730 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.861711    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"} err="failed to get container status \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": rpc error: code = NotFound desc = could not find container \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": container with ID starting with 0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447 not found: ID does not exist"
Aug 04 20:44:11.861797 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.861732    2565 scope.go:117] "RemoveContainer" containerID="4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"
Aug 04 20:44:11.861957 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.861940    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"} err="failed to get container status \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": rpc error: code = NotFound desc = could not find container \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": container with ID starting with 4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1 not found: ID does not exist"
Aug 04 20:44:11.862007 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.861958    2565 scope.go:117] "RemoveContainer" containerID="8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"
Aug 04 20:44:11.862170 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.862148    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"} err="failed to get container status \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": rpc error: code = NotFound desc = could not find container \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": container with ID starting with 8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9 not found: ID does not exist"
Aug 04 20:44:11.862217 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.862170    2565 scope.go:117] "RemoveContainer" containerID="00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"
Aug 04 20:44:11.862393 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.862374    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"} err="failed to get container status \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": rpc error: code = NotFound desc = could not find container \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": container with ID starting with 00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64 not found: ID does not exist"
Aug 04 20:44:11.862433 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.862394    2565 scope.go:117] "RemoveContainer" containerID="20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"
Aug 04 20:44:11.862611 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.862593    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"} err="failed to get container status \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": rpc error: code = NotFound desc = could not find container \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": container with ID starting with 20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee not found: ID does not exist"
Aug 04 20:44:11.862647 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.862613    2565 scope.go:117] "RemoveContainer" containerID="559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"
Aug 04 20:44:11.862844 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.862825    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"} err="failed to get container status \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": rpc error: code = NotFound desc = could not find container \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": container with ID starting with 559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32 not found: ID does not exist"
Aug 04 20:44:11.862844 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.862843    2565 scope.go:117] "RemoveContainer" containerID="4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"
Aug 04 20:44:11.863052 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.863035    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"} err="failed to get container status \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": rpc error: code = NotFound desc = could not find container \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": container with ID starting with 4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157 not found: ID does not exist"
Aug 04 20:44:11.863101 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.863053    2565 scope.go:117] "RemoveContainer" containerID="0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"
Aug 04 20:44:11.863319 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.863293    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"} err="failed to get container status \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": rpc error: code = NotFound desc = could not find container \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": container with ID starting with 0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447 not found: ID does not exist"
Aug 04 20:44:11.863319 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.863320    2565 scope.go:117] "RemoveContainer" containerID="4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"
Aug 04 20:44:11.863625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.863600    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"} err="failed to get container status \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": rpc error: code = NotFound desc = could not find container \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": container with ID starting with 4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1 not found: ID does not exist"
Aug 04 20:44:11.863625 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.863625    2565 scope.go:117] "RemoveContainer" containerID="8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"
Aug 04 20:44:11.863883 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.863864    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"} err="failed to get container status \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": rpc error: code = NotFound desc = could not find container \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": container with ID starting with 8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9 not found: ID does not exist"
Aug 04 20:44:11.863883 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.863883    2565 scope.go:117] "RemoveContainer" containerID="00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"
Aug 04 20:44:11.864209 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.864172    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"} err="failed to get container status \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": rpc error: code = NotFound desc = could not find container \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": container with ID starting with 00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64 not found: ID does not exist"
Aug 04 20:44:11.864293 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.864221    2565 scope.go:117] "RemoveContainer" containerID="20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"
Aug 04 20:44:11.864520 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.864501    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"} err="failed to get container status \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": rpc error: code = NotFound desc = could not find container \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": container with ID starting with 20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee not found: ID does not exist"
Aug 04 20:44:11.864577 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.864520    2565 scope.go:117] "RemoveContainer" containerID="559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"
Aug 04 20:44:11.864806 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.864784    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"} err="failed to get container status \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": rpc error: code = NotFound desc = could not find container \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": container with ID starting with 559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32 not found: ID does not exist"
Aug 04 20:44:11.864806 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.864805    2565 scope.go:117] "RemoveContainer" containerID="4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"
Aug 04 20:44:11.865060 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.865041    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"} err="failed to get container status \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": rpc error: code = NotFound desc = could not find container \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": container with ID starting with 4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157 not found: ID does not exist"
Aug 04 20:44:11.865115 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.865061    2565 scope.go:117] "RemoveContainer" containerID="0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"
Aug 04 20:44:11.865300 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.865281    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"} err="failed to get container status \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": rpc error: code = NotFound desc = could not find container \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": container with ID starting with 0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447 not found: ID does not exist"
Aug 04 20:44:11.865356 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.865301    2565 scope.go:117] "RemoveContainer" containerID="4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"
Aug 04 20:44:11.865559 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.865528    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"} err="failed to get container status \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": rpc error: code = NotFound desc = could not find container \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": container with ID starting with 4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1 not found: ID does not exist"
Aug 04 20:44:11.865609 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.865560    2565 scope.go:117] "RemoveContainer" containerID="8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"
Aug 04 20:44:11.865788 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.865768    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"} err="failed to get container status \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": rpc error: code = NotFound desc = could not find container \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": container with ID starting with 8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9 not found: ID does not exist"
Aug 04 20:44:11.865858 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.865790    2565 scope.go:117] "RemoveContainer" containerID="00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"
Aug 04 20:44:11.866038 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.866022    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"} err="failed to get container status \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": rpc error: code = NotFound desc = could not find container \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": container with ID starting with 00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64 not found: ID does not exist"
Aug 04 20:44:11.866086 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.866038    2565 scope.go:117] "RemoveContainer" containerID="20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"
Aug 04 20:44:11.866224 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.866210    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.866327 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.866209    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"} err="failed to get container status \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": rpc error: code = NotFound desc = could not find container \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": container with ID starting with 20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee not found: ID does not exist"
Aug 04 20:44:11.866504 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.866329    2565 scope.go:117] "RemoveContainer" containerID="559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"
Aug 04 20:44:11.866759 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.866729    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"} err="failed to get container status \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": rpc error: code = NotFound desc = could not find container \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": container with ID starting with 559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32 not found: ID does not exist"
Aug 04 20:44:11.866759 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.866759    2565 scope.go:117] "RemoveContainer" containerID="4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"
Aug 04 20:44:11.867029 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.867006    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"} err="failed to get container status \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": rpc error: code = NotFound desc = could not find container \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": container with ID starting with 4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157 not found: ID does not exist"
Aug 04 20:44:11.867078 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.867031    2565 scope.go:117] "RemoveContainer" containerID="0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"
Aug 04 20:44:11.867252 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.867232    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"} err="failed to get container status \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": rpc error: code = NotFound desc = could not find container \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": container with ID starting with 0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447 not found: ID does not exist"
Aug 04 20:44:11.867304 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.867252    2565 scope.go:117] "RemoveContainer" containerID="4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"
Aug 04 20:44:11.867509 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.867488    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"} err="failed to get container status \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": rpc error: code = NotFound desc = could not find container \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": container with ID starting with 4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1 not found: ID does not exist"
Aug 04 20:44:11.867509 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.867507    2565 scope.go:117] "RemoveContainer" containerID="8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"
Aug 04 20:44:11.867746 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.867729    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9"} err="failed to get container status \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": rpc error: code = NotFound desc = could not find container \"8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9\": container with ID starting with 8c7314b543a3cccc4699834655492a231b317d9926c61c2357febc230cbc43a9 not found: ID does not exist"
Aug 04 20:44:11.867796 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.867747    2565 scope.go:117] "RemoveContainer" containerID="00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"
Aug 04 20:44:11.867957 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.867939    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64"} err="failed to get container status \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": rpc error: code = NotFound desc = could not find container \"00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64\": container with ID starting with 00299725f362c892edd64543c4916adb1ceabba5d2f12401bc9579b9daecff64 not found: ID does not exist"
Aug 04 20:44:11.868003 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.867961    2565 scope.go:117] "RemoveContainer" containerID="20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"
Aug 04 20:44:11.868230 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.868211    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee"} err="failed to get container status \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": rpc error: code = NotFound desc = could not find container \"20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee\": container with ID starting with 20364c54f8a432d81a28b8337c50de90b2a46dba56e338ede312a6cccdde28ee not found: ID does not exist"
Aug 04 20:44:11.868230 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.868229    2565 scope.go:117] "RemoveContainer" containerID="559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"
Aug 04 20:44:11.868475 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.868458    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32"} err="failed to get container status \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": rpc error: code = NotFound desc = could not find container \"559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32\": container with ID starting with 559c93b24dec595642cf10cfbbafae1c931db4499c7bac9b04aa482ed8646f32 not found: ID does not exist"
Aug 04 20:44:11.868475 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.868476    2565 scope.go:117] "RemoveContainer" containerID="4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"
Aug 04 20:44:11.868641 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.868623    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-17o59lmt02unb\""
Aug 04 20:44:11.868692 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.868628    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Aug 04 20:44:11.868735 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.868704    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157"} err="failed to get container status \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": rpc error: code = NotFound desc = could not find container \"4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157\": container with ID starting with 4d685f013d3682612c6146c201e27ba41aba59e462b2dd26706835f2ddc7b157 not found: ID does not exist"
Aug 04 20:44:11.868735 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.868730    2565 scope.go:117] "RemoveContainer" containerID="0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"
Aug 04 20:44:11.868986 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.868962    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447"} err="failed to get container status \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": rpc error: code = NotFound desc = could not find container \"0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447\": container with ID starting with 0bc2b447ad4bb4f10dd0ec46f03e3edbb8d65a42865a763b54ecc049133fe447 not found: ID does not exist"
Aug 04 20:44:11.869079 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869004    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Aug 04 20:44:11.869079 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869011    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Aug 04 20:44:11.869079 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869007    2565 scope.go:117] "RemoveContainer" containerID="4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"
Aug 04 20:44:11.869242 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869109    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Aug 04 20:44:11.869242 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869111    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8p8xp\""
Aug 04 20:44:11.869362 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869316    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1"} err="failed to get container status \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": rpc error: code = NotFound desc = could not find container \"4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1\": container with ID starting with 4cf0cafcc8c57d6a394184449514091bc1498e7fb21d61cb9d4bae9007c742d1 not found: ID does not exist"
Aug 04 20:44:11.869362 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869323    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Aug 04 20:44:11.869486 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869466    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Aug 04 20:44:11.869486 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869476    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Aug 04 20:44:11.869589 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869577    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Aug 04 20:44:11.869630 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869596    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Aug 04 20:44:11.869955 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.869939    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Aug 04 20:44:11.870122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.870097    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Aug 04 20:44:11.870122 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.870116    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Aug 04 20:44:11.870248 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.870162    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Aug 04 20:44:11.872529 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.872510    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Aug 04 20:44:11.875554 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.875536    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Aug 04 20:44:11.881021 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.881003    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 20:44:11.926659 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.926636    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.926780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.926670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.926780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.926696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.926780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.926720    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.926917 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.926796    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.926917 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.926821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sn54\" (UniqueName: \"kubernetes.io/projected/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-kube-api-access-8sn54\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927017 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.926929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927017 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.926958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927112 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.927045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927112 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.927101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927211 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.927127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927211 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.927155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927313 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.927283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927395 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.927322    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927488 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.927413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927602 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.927569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927908 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.927841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927993 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.927910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.927993 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.927981    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:11.928105 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:11.928025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-config\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.028640 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.028608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.028640 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.028642    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.028840 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.028663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.028840 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.028681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.028840 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.028721    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.028976 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.028910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sn54\" (UniqueName: \"kubernetes.io/projected/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-kube-api-access-8sn54\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.028976 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.028952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029323    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-config\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.029940 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.029716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.030862 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.030338    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.030862 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.030378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.031791 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.031763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.031908 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.031816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.031908 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.031899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.032030 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.031986    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.032371 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.032201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-config\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.032923 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.032690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.032923 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.032818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.032923 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.032880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.033701 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.033673    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.034081 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.034058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.034152 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.034101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.034214 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.034197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.034519 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.034502    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.034958 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.034937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.037577 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.037559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sn54\" (UniqueName: \"kubernetes.io/projected/9b4cbfb6-1cb8-4896-9adb-1e89abf249a8-kube-api-access-8sn54\") pod \"prometheus-k8s-0\" (UID: \"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.176676 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.176643    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:44:12.345389 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.345258    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 20:44:12.347876 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:44:12.347845    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4cbfb6_1cb8_4896_9adb_1e89abf249a8.slice/crio-61e89161ade04c48442496b5fb3983b4f17d513e21b6473e3fdacf074f09768f WatchSource:0}: Error finding container 61e89161ade04c48442496b5fb3983b4f17d513e21b6473e3fdacf074f09768f: Status 404 returned error can't find the container with id 61e89161ade04c48442496b5fb3983b4f17d513e21b6473e3fdacf074f09768f
Aug 04 20:44:12.811998 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.811958    2565 generic.go:356] "Generic (PLEG): container finished" podID="9b4cbfb6-1cb8-4896-9adb-1e89abf249a8" containerID="2e6751e1724ce7f809e8976dd955cfc6addaf31965ada738b0a6a88aaff76022" exitCode=0
Aug 04 20:44:12.812482 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.812044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8","Type":"ContainerDied","Data":"2e6751e1724ce7f809e8976dd955cfc6addaf31965ada738b0a6a88aaff76022"}
Aug 04 20:44:12.812482 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:12.812150    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8","Type":"ContainerStarted","Data":"61e89161ade04c48442496b5fb3983b4f17d513e21b6473e3fdacf074f09768f"}
Aug 04 20:44:13.759804 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.759773    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7efa6de3-c112-43e7-b24c-e383e7112126" path="/var/lib/kubelet/pods/7efa6de3-c112-43e7-b24c-e383e7112126/volumes"
Aug 04 20:44:13.819301 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.819266    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n" event={"ID":"c91f55b7-fdea-4528-bcc7-3a283561bab7","Type":"ContainerStarted","Data":"55ea5faeb7a9693228a6136ad9e44aa56a580c40c1fb8c00b4102fed35c96e65"}
Aug 04 20:44:13.819712 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.819335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n" event={"ID":"c91f55b7-fdea-4528-bcc7-3a283561bab7","Type":"ContainerStarted","Data":"c76529c40e7145b4c0aa45af31abf40068140865f2b85ccb0e18391c268427ac"}
Aug 04 20:44:13.819712 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.819383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n" event={"ID":"c91f55b7-fdea-4528-bcc7-3a283561bab7","Type":"ContainerStarted","Data":"c22600591447434c148c19a5e6a66aab0f00d47ea9a3bea13710f56275472479"}
Aug 04 20:44:13.822524 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.822494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8","Type":"ContainerStarted","Data":"e8aeea0f303348f9f138a4c8a35075ba1dab6e230513bba764e3868e36a7e481"}
Aug 04 20:44:13.822656 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.822549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8","Type":"ContainerStarted","Data":"ce8ce54bc15cbd11dfdce72a7e703aa9ef397c457256d35402017f1e132b5cd5"}
Aug 04 20:44:13.822656 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.822588    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8","Type":"ContainerStarted","Data":"b67e2dad74719825c3d3075f45c3da062235fcdef0a01269b1bc859292b83753"}
Aug 04 20:44:13.822656 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.822612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8","Type":"ContainerStarted","Data":"b15fe84f6f6ef8c02ca349eec2bfad5c22d651f253d08f5c89171b342e75bbd2"}
Aug 04 20:44:13.822656 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.822640    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8","Type":"ContainerStarted","Data":"ceb8cea31e9ca8300f0710b44f5dd86ea54cdf3fe47820ba1e4487eb98539346"}
Aug 04 20:44:13.822799 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.822663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b4cbfb6-1cb8-4896-9adb-1e89abf249a8","Type":"ContainerStarted","Data":"6a8d15f809e6bdc350ac4d2100010f8f56b0fec61200292ece968ef4513153e0"}
Aug 04 20:44:13.845036 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.844981    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-649c45fdd-rjc6n" podStartSLOduration=2.183271171 podStartE2EDuration="3.844963109s" totalImagesPullingTime="1.661691938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:44:13.842419923 +0000 UTC m=+228.594525899" watchObservedRunningTime="2026-08-04 20:44:13.844963109 +0000 UTC m=+228.597069088"
Aug 04 20:44:13.873827 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:13.873784    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.855044162 podStartE2EDuration="2.873768174s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.724012ms" isStatefulPod=false podCreationTimestamp="2026-08-04 20:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:44:13.872975134 +0000 UTC m=+228.625081112" watchObservedRunningTime="2026-08-04 20:44:13.873768174 +0000 UTC m=+228.625874154"
Aug 04 20:44:17.177502 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:44:17.177467    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:45:12.177132 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:45:12.177096    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:45:12.192642 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:45:12.192621    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:45:13.065207 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:45:13.065182    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 20:45:25.668466 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:45:25.668434    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 20:45:25.675695 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:45:25.675671    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 20:45:25.678420 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:45:25.678398    2565 kubelet.go:1648] "Image garbage collection succeeded"
Aug 04 20:47:21.615053 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.615020    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fhtrr"]
Aug 04 20:47:21.618335 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.618314    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fhtrr"
Aug 04 20:47:21.621190 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.621172    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xx5xk\""
Aug 04 20:47:21.621309 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.621206    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Aug 04 20:47:21.621309 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.621271    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Aug 04 20:47:21.627862 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.627839    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fhtrr"]
Aug 04 20:47:21.769041 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.769011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7bfeb52-ff80-40ba-bf35-eda17b570e83-bound-sa-token\") pod \"cert-manager-55b4d954d5-fhtrr\" (UID: \"b7bfeb52-ff80-40ba-bf35-eda17b570e83\") " pod="cert-manager/cert-manager-55b4d954d5-fhtrr"
Aug 04 20:47:21.769208 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.769058    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlsh\" (UniqueName: \"kubernetes.io/projected/b7bfeb52-ff80-40ba-bf35-eda17b570e83-kube-api-access-wqlsh\") pod \"cert-manager-55b4d954d5-fhtrr\" (UID: \"b7bfeb52-ff80-40ba-bf35-eda17b570e83\") " pod="cert-manager/cert-manager-55b4d954d5-fhtrr"
Aug 04 20:47:21.870294 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.870209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlsh\" (UniqueName: \"kubernetes.io/projected/b7bfeb52-ff80-40ba-bf35-eda17b570e83-kube-api-access-wqlsh\") pod \"cert-manager-55b4d954d5-fhtrr\" (UID: \"b7bfeb52-ff80-40ba-bf35-eda17b570e83\") " pod="cert-manager/cert-manager-55b4d954d5-fhtrr"
Aug 04 20:47:21.870457 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.870297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7bfeb52-ff80-40ba-bf35-eda17b570e83-bound-sa-token\") pod \"cert-manager-55b4d954d5-fhtrr\" (UID: \"b7bfeb52-ff80-40ba-bf35-eda17b570e83\") " pod="cert-manager/cert-manager-55b4d954d5-fhtrr"
Aug 04 20:47:21.878059 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.878034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7bfeb52-ff80-40ba-bf35-eda17b570e83-bound-sa-token\") pod \"cert-manager-55b4d954d5-fhtrr\" (UID: \"b7bfeb52-ff80-40ba-bf35-eda17b570e83\") " pod="cert-manager/cert-manager-55b4d954d5-fhtrr"
Aug 04 20:47:21.878232 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.878210    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlsh\" (UniqueName: \"kubernetes.io/projected/b7bfeb52-ff80-40ba-bf35-eda17b570e83-kube-api-access-wqlsh\") pod \"cert-manager-55b4d954d5-fhtrr\" (UID: \"b7bfeb52-ff80-40ba-bf35-eda17b570e83\") " pod="cert-manager/cert-manager-55b4d954d5-fhtrr"
Aug 04 20:47:21.939047 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:21.939015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fhtrr"
Aug 04 20:47:22.065289 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:22.065264    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fhtrr"]
Aug 04 20:47:22.067716 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:47:22.067692    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7bfeb52_ff80_40ba_bf35_eda17b570e83.slice/crio-5d0754d1f47cd1b91b9199f327c1abd2a1578ad0db8d5ce21bbf4c21ca2b55ef WatchSource:0}: Error finding container 5d0754d1f47cd1b91b9199f327c1abd2a1578ad0db8d5ce21bbf4c21ca2b55ef: Status 404 returned error can't find the container with id 5d0754d1f47cd1b91b9199f327c1abd2a1578ad0db8d5ce21bbf4c21ca2b55ef
Aug 04 20:47:22.069645 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:22.069627    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 20:47:22.566777 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:22.566743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fhtrr" event={"ID":"b7bfeb52-ff80-40ba-bf35-eda17b570e83","Type":"ContainerStarted","Data":"5d0754d1f47cd1b91b9199f327c1abd2a1578ad0db8d5ce21bbf4c21ca2b55ef"}
Aug 04 20:47:25.583073 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:25.583036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fhtrr" event={"ID":"b7bfeb52-ff80-40ba-bf35-eda17b570e83","Type":"ContainerStarted","Data":"1ae725574e2fc4a123aad913b3a39a6b5bd69e591ad5ea71c8c187ea9b588c2a"}
Aug 04 20:47:25.601557 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:25.601508    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-fhtrr" podStartSLOduration=1.474536579 podStartE2EDuration="4.60149313s" totalImagesPullingTime="3.126956551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:47:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:47:25.600398298 +0000 UTC m=+420.352504279" watchObservedRunningTime="2026-08-04 20:47:25.60149313 +0000 UTC m=+420.353599106"
Aug 04 20:47:46.799636 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.799599    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"]
Aug 04 20:47:46.803176 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.803158    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:46.806433 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.806414    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Aug 04 20:47:46.806433 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.806428    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Aug 04 20:47:46.807337 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.807318    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2p2tb\""
Aug 04 20:47:46.807422 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.807337    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Aug 04 20:47:46.807574 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.807559    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Aug 04 20:47:46.807659 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.807644    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Aug 04 20:47:46.821749 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.821728    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"]
Aug 04 20:47:46.979566 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.979535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-metrics-cert\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:46.979748 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.979591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmtkm\" (UniqueName: \"kubernetes.io/projected/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-kube-api-access-fmtkm\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:46.979748 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.979701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-manager-config\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:46.979849 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:46.979750    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-cert\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:47.080772 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.080689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-manager-config\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:47.080772 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.080726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-cert\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:47.080772 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.080756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-metrics-cert\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:47.081042 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.080930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmtkm\" (UniqueName: \"kubernetes.io/projected/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-kube-api-access-fmtkm\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:47.081469 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.081443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-manager-config\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:47.083143 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.083125    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-cert\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:47.083199 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.083181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-metrics-cert\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:47.103678 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.103654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmtkm\" (UniqueName: \"kubernetes.io/projected/4108f0f3-7453-4ff5-b8c6-c6fc838c355e-kube-api-access-fmtkm\") pod \"lws-controller-manager-7564c45d7d-r2jps\" (UID: \"4108f0f3-7453-4ff5-b8c6-c6fc838c355e\") " pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:47.112433 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.112413    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:47.300158 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.300118    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"]
Aug 04 20:47:47.304429 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:47:47.304390    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4108f0f3_7453_4ff5_b8c6_c6fc838c355e.slice/crio-b996804a02fd40d392fb47c2b0efa08111600c448f03ff1fec489e7bfde8b9f0 WatchSource:0}: Error finding container b996804a02fd40d392fb47c2b0efa08111600c448f03ff1fec489e7bfde8b9f0: Status 404 returned error can't find the container with id b996804a02fd40d392fb47c2b0efa08111600c448f03ff1fec489e7bfde8b9f0
Aug 04 20:47:47.617229 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.617195    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"]
Aug 04 20:47:47.621853 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.621835    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.624587 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.624566    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4mstc\""
Aug 04 20:47:47.624972 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.624956    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Aug 04 20:47:47.625057 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.624968    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Aug 04 20:47:47.625057 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.625039    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Aug 04 20:47:47.625989 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.625973    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Aug 04 20:47:47.639983 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.639963    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"]
Aug 04 20:47:47.672551 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.672521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps" event={"ID":"4108f0f3-7453-4ff5-b8c6-c6fc838c355e","Type":"ContainerStarted","Data":"b996804a02fd40d392fb47c2b0efa08111600c448f03ff1fec489e7bfde8b9f0"}
Aug 04 20:47:47.788275 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.788250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ce09fc4-54bf-4662-a8a1-f73733026846-manifests\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.788450 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.788304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ce09fc4-54bf-4662-a8a1-f73733026846-tmp\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.788450 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.788385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ce09fc4-54bf-4662-a8a1-f73733026846-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.788450 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.788411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ce09fc4-54bf-4662-a8a1-f73733026846-webhook-cert\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.788568 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.788507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4jn\" (UniqueName: \"kubernetes.io/projected/5ce09fc4-54bf-4662-a8a1-f73733026846-kube-api-access-lc4jn\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.889908 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.889842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4jn\" (UniqueName: \"kubernetes.io/projected/5ce09fc4-54bf-4662-a8a1-f73733026846-kube-api-access-lc4jn\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.889908 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.889881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ce09fc4-54bf-4662-a8a1-f73733026846-manifests\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.890376 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.889911    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ce09fc4-54bf-4662-a8a1-f73733026846-tmp\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.890376 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.889938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ce09fc4-54bf-4662-a8a1-f73733026846-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.890376 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.889954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ce09fc4-54bf-4662-a8a1-f73733026846-webhook-cert\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.890376 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.890248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ce09fc4-54bf-4662-a8a1-f73733026846-manifests\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.892161 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.892140    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ce09fc4-54bf-4662-a8a1-f73733026846-tmp\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.892414 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.892398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ce09fc4-54bf-4662-a8a1-f73733026846-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.892466 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.892455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ce09fc4-54bf-4662-a8a1-f73733026846-webhook-cert\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.898419 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.898401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4jn\" (UniqueName: \"kubernetes.io/projected/5ce09fc4-54bf-4662-a8a1-f73733026846-kube-api-access-lc4jn\") pod \"opendatahub-operator-controller-manager-6889885b4f-sgccm\" (UID: \"5ce09fc4-54bf-4662-a8a1-f73733026846\") " pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:47.932497 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:47.932474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:48.100260 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:48.100237    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"]
Aug 04 20:47:48.102909 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:47:48.102878    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce09fc4_54bf_4662_a8a1_f73733026846.slice/crio-8a8fba239fde72414ba67d04eedb6de199b1a37988cdf5cb16af1ba4b090746f WatchSource:0}: Error finding container 8a8fba239fde72414ba67d04eedb6de199b1a37988cdf5cb16af1ba4b090746f: Status 404 returned error can't find the container with id 8a8fba239fde72414ba67d04eedb6de199b1a37988cdf5cb16af1ba4b090746f
Aug 04 20:47:48.681522 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:48.681440    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm" event={"ID":"5ce09fc4-54bf-4662-a8a1-f73733026846","Type":"ContainerStarted","Data":"8a8fba239fde72414ba67d04eedb6de199b1a37988cdf5cb16af1ba4b090746f"}
Aug 04 20:47:52.701703 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:52.701658    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm" event={"ID":"5ce09fc4-54bf-4662-a8a1-f73733026846","Type":"ContainerStarted","Data":"8fcb8c6c1504105b9719dae54be535c3b4ed6cd958a1f2a9b5dd56c152905738"}
Aug 04 20:47:52.703029 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:52.703003    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps" event={"ID":"4108f0f3-7453-4ff5-b8c6-c6fc838c355e","Type":"ContainerStarted","Data":"9b2ecaa7bff336bcbb53205353022cc45da4f2da52ad9c909e293f286cda8173"}
Aug 04 20:47:52.703210 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:52.703193    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:47:52.749555 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:52.749511    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps" podStartSLOduration=2.3696751799999998 podStartE2EDuration="6.749494913s" totalImagesPullingTime="4.379819733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:47:52.748336895 +0000 UTC m=+447.500442885" watchObservedRunningTime="2026-08-04 20:47:52.749494913 +0000 UTC m=+447.501600889"
Aug 04 20:47:53.709694 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:53.709660    2565 generic.go:356] "Generic (PLEG): container finished" podID="5ce09fc4-54bf-4662-a8a1-f73733026846" containerID="8fcb8c6c1504105b9719dae54be535c3b4ed6cd958a1f2a9b5dd56c152905738" exitCode=0
Aug 04 20:47:53.710098 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:53.709750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm" event={"ID":"5ce09fc4-54bf-4662-a8a1-f73733026846","Type":"ContainerDied","Data":"8fcb8c6c1504105b9719dae54be535c3b4ed6cd958a1f2a9b5dd56c152905738"}
Aug 04 20:47:54.716200 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:54.716169    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm" event={"ID":"5ce09fc4-54bf-4662-a8a1-f73733026846","Type":"ContainerStarted","Data":"218572faab70df5f8818b1aa82908605a407cafc918bbf20fbd45e4d0de3923c"}
Aug 04 20:47:54.716618 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:54.716300    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:47:54.740811 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:47:54.740762    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm" podStartSLOduration=2.772272202 podStartE2EDuration="7.740744843s" totalImagesPullingTime="3.922893275s" totalInitContainerRuntime="1.045579366s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:47:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:47:54.739540844 +0000 UTC m=+449.491646811" watchObservedRunningTime="2026-08-04 20:47:54.740744843 +0000 UTC m=+449.492850820"
Aug 04 20:48:03.712738 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:03.712708    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7564c45d7d-r2jps"
Aug 04 20:48:05.723296 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:05.723265    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6889885b4f-sgccm"
Aug 04 20:48:07.981037 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:07.981006    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"]
Aug 04 20:48:07.990588 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:07.990562    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"
Aug 04 20:48:07.993082 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:07.993060    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Aug 04 20:48:07.994110 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:07.994084    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Aug 04 20:48:07.994223 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:07.994118    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Aug 04 20:48:07.994223 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:07.994090    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zxkm5\""
Aug 04 20:48:07.994223 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:07.994191    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Aug 04 20:48:07.996821 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:07.996800    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"]
Aug 04 20:48:08.064686 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.064659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a092ad57-c542-47b9-9f78-30f22c3f6f9d-tmp\") pod \"kube-auth-proxy-586b55cbcc-tc95q\" (UID: \"a092ad57-c542-47b9-9f78-30f22c3f6f9d\") " pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"
Aug 04 20:48:08.064889 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.064704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a092ad57-c542-47b9-9f78-30f22c3f6f9d-tls-certs\") pod \"kube-auth-proxy-586b55cbcc-tc95q\" (UID: \"a092ad57-c542-47b9-9f78-30f22c3f6f9d\") " pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"
Aug 04 20:48:08.064889 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.064813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96tn\" (UniqueName: \"kubernetes.io/projected/a092ad57-c542-47b9-9f78-30f22c3f6f9d-kube-api-access-r96tn\") pod \"kube-auth-proxy-586b55cbcc-tc95q\" (UID: \"a092ad57-c542-47b9-9f78-30f22c3f6f9d\") " pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"
Aug 04 20:48:08.165403 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.165370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96tn\" (UniqueName: \"kubernetes.io/projected/a092ad57-c542-47b9-9f78-30f22c3f6f9d-kube-api-access-r96tn\") pod \"kube-auth-proxy-586b55cbcc-tc95q\" (UID: \"a092ad57-c542-47b9-9f78-30f22c3f6f9d\") " pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"
Aug 04 20:48:08.165559 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.165439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a092ad57-c542-47b9-9f78-30f22c3f6f9d-tmp\") pod \"kube-auth-proxy-586b55cbcc-tc95q\" (UID: \"a092ad57-c542-47b9-9f78-30f22c3f6f9d\") " pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"
Aug 04 20:48:08.165559 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.165470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a092ad57-c542-47b9-9f78-30f22c3f6f9d-tls-certs\") pod \"kube-auth-proxy-586b55cbcc-tc95q\" (UID: \"a092ad57-c542-47b9-9f78-30f22c3f6f9d\") " pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"
Aug 04 20:48:08.167821 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.167794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a092ad57-c542-47b9-9f78-30f22c3f6f9d-tmp\") pod \"kube-auth-proxy-586b55cbcc-tc95q\" (UID: \"a092ad57-c542-47b9-9f78-30f22c3f6f9d\") " pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"
Aug 04 20:48:08.167944 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.167884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a092ad57-c542-47b9-9f78-30f22c3f6f9d-tls-certs\") pod \"kube-auth-proxy-586b55cbcc-tc95q\" (UID: \"a092ad57-c542-47b9-9f78-30f22c3f6f9d\") " pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"
Aug 04 20:48:08.178712 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.178682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96tn\" (UniqueName: \"kubernetes.io/projected/a092ad57-c542-47b9-9f78-30f22c3f6f9d-kube-api-access-r96tn\") pod \"kube-auth-proxy-586b55cbcc-tc95q\" (UID: \"a092ad57-c542-47b9-9f78-30f22c3f6f9d\") " pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"
Aug 04 20:48:08.302468 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.302385    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"
Aug 04 20:48:08.452743 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.452719    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q"]
Aug 04 20:48:08.454201 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:48:08.454172    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda092ad57_c542_47b9_9f78_30f22c3f6f9d.slice/crio-b95a16259566119a8344143588272de8af55729dfeddc115a7b6ce3069ac4da0 WatchSource:0}: Error finding container b95a16259566119a8344143588272de8af55729dfeddc115a7b6ce3069ac4da0: Status 404 returned error can't find the container with id b95a16259566119a8344143588272de8af55729dfeddc115a7b6ce3069ac4da0
Aug 04 20:48:08.780626 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:08.780595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q" event={"ID":"a092ad57-c542-47b9-9f78-30f22c3f6f9d","Type":"ContainerStarted","Data":"b95a16259566119a8344143588272de8af55729dfeddc115a7b6ce3069ac4da0"}
Aug 04 20:48:11.796691 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:11.796648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q" event={"ID":"a092ad57-c542-47b9-9f78-30f22c3f6f9d","Type":"ContainerStarted","Data":"60b110f0fc8dad8b285794f00dc621dbd031b6b0e1b8825ef7878363b4495bbb"}
Aug 04 20:48:11.818086 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:48:11.818036    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-586b55cbcc-tc95q" podStartSLOduration=1.85419479 podStartE2EDuration="4.818019937s" totalImagesPullingTime="2.963825147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:48:11.816072 +0000 UTC m=+466.568177978" watchObservedRunningTime="2026-08-04 20:48:11.818019937 +0000 UTC m=+466.570125916"
Aug 04 20:49:37.797058 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.796971    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"]
Aug 04 20:49:37.800444 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.800421    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:37.803199 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.803178    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Aug 04 20:49:37.803391 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.803249    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Aug 04 20:49:37.804460 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.804439    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Aug 04 20:49:37.804460 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.804453    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qht4l\""
Aug 04 20:49:37.804594 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.804444    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Aug 04 20:49:37.809315 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.809293    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"]
Aug 04 20:49:37.833998 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.833978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d83ed8a6-f4b3-4788-8f51-546959ce78ed-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-6tcdn\" (UID: \"d83ed8a6-f4b3-4788-8f51-546959ce78ed\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:37.834108 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.834038    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvsc9\" (UniqueName: \"kubernetes.io/projected/d83ed8a6-f4b3-4788-8f51-546959ce78ed-kube-api-access-jvsc9\") pod \"kuadrant-console-plugin-598f86468b-6tcdn\" (UID: \"d83ed8a6-f4b3-4788-8f51-546959ce78ed\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:37.834150 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.834099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d83ed8a6-f4b3-4788-8f51-546959ce78ed-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-6tcdn\" (UID: \"d83ed8a6-f4b3-4788-8f51-546959ce78ed\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:37.934622 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.934592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsc9\" (UniqueName: \"kubernetes.io/projected/d83ed8a6-f4b3-4788-8f51-546959ce78ed-kube-api-access-jvsc9\") pod \"kuadrant-console-plugin-598f86468b-6tcdn\" (UID: \"d83ed8a6-f4b3-4788-8f51-546959ce78ed\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:37.934739 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.934630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d83ed8a6-f4b3-4788-8f51-546959ce78ed-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-6tcdn\" (UID: \"d83ed8a6-f4b3-4788-8f51-546959ce78ed\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:37.934802 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.934763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d83ed8a6-f4b3-4788-8f51-546959ce78ed-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-6tcdn\" (UID: \"d83ed8a6-f4b3-4788-8f51-546959ce78ed\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:37.934910 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:49:37.934895    2565 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Aug 04 20:49:37.934959 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:49:37.934949    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83ed8a6-f4b3-4788-8f51-546959ce78ed-plugin-serving-cert podName:d83ed8a6-f4b3-4788-8f51-546959ce78ed nodeName:}" failed. No retries permitted until 2026-08-04 20:49:38.434932772 +0000 UTC m=+553.187038729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d83ed8a6-f4b3-4788-8f51-546959ce78ed-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-6tcdn" (UID: "d83ed8a6-f4b3-4788-8f51-546959ce78ed") : secret "plugin-serving-cert" not found
Aug 04 20:49:37.935230 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.935213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d83ed8a6-f4b3-4788-8f51-546959ce78ed-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-6tcdn\" (UID: \"d83ed8a6-f4b3-4788-8f51-546959ce78ed\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:37.944124 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:37.944102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvsc9\" (UniqueName: \"kubernetes.io/projected/d83ed8a6-f4b3-4788-8f51-546959ce78ed-kube-api-access-jvsc9\") pod \"kuadrant-console-plugin-598f86468b-6tcdn\" (UID: \"d83ed8a6-f4b3-4788-8f51-546959ce78ed\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:38.439244 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:38.439211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d83ed8a6-f4b3-4788-8f51-546959ce78ed-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-6tcdn\" (UID: \"d83ed8a6-f4b3-4788-8f51-546959ce78ed\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:38.441626 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:38.441597    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d83ed8a6-f4b3-4788-8f51-546959ce78ed-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-6tcdn\" (UID: \"d83ed8a6-f4b3-4788-8f51-546959ce78ed\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:38.711378 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:38.711258    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"
Aug 04 20:49:38.862135 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:38.862099    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn"]
Aug 04 20:49:38.865834 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:49:38.865807    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd83ed8a6_f4b3_4788_8f51_546959ce78ed.slice/crio-9ae2dfbabcaf1bdd2422520f533743051869802588a7f24d9beaf0d45986c5ad WatchSource:0}: Error finding container 9ae2dfbabcaf1bdd2422520f533743051869802588a7f24d9beaf0d45986c5ad: Status 404 returned error can't find the container with id 9ae2dfbabcaf1bdd2422520f533743051869802588a7f24d9beaf0d45986c5ad
Aug 04 20:49:39.188641 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:39.188604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn" event={"ID":"d83ed8a6-f4b3-4788-8f51-546959ce78ed","Type":"ContainerStarted","Data":"9ae2dfbabcaf1bdd2422520f533743051869802588a7f24d9beaf0d45986c5ad"}
Aug 04 20:49:44.220978 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:44.220940    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn" event={"ID":"d83ed8a6-f4b3-4788-8f51-546959ce78ed","Type":"ContainerStarted","Data":"ce22333c9c52b0c6034805a3cd8e931dab3b7c1fbb9433ef7f37d15afe9d71f5"}
Aug 04 20:49:44.236401 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:44.236311    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-6tcdn" podStartSLOduration=2.480402384 podStartE2EDuration="7.236296076s" totalImagesPullingTime="4.755893692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:49:44.235704628 +0000 UTC m=+558.987810627" watchObservedRunningTime="2026-08-04 20:49:44.236296076 +0000 UTC m=+558.988402052"
Aug 04 20:49:48.161164 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.161132    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"]
Aug 04 20:49:48.164458 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.164436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:49:48.167455 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.167430    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n66wc\""
Aug 04 20:49:48.185074 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.185046    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"]
Aug 04 20:49:48.227884 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.227855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b651297-b930-4d64-9e35-e5632ba89357-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-h5j2k\" (UID: \"1b651297-b930-4d64-9e35-e5632ba89357\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:49:48.228057 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.227908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwkp\" (UniqueName: \"kubernetes.io/projected/1b651297-b930-4d64-9e35-e5632ba89357-kube-api-access-hkwkp\") pod \"kuadrant-operator-controller-manager-58fc56757c-h5j2k\" (UID: \"1b651297-b930-4d64-9e35-e5632ba89357\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:49:48.328459 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.328426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b651297-b930-4d64-9e35-e5632ba89357-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-h5j2k\" (UID: \"1b651297-b930-4d64-9e35-e5632ba89357\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:49:48.328617 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.328471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwkp\" (UniqueName: \"kubernetes.io/projected/1b651297-b930-4d64-9e35-e5632ba89357-kube-api-access-hkwkp\") pod \"kuadrant-operator-controller-manager-58fc56757c-h5j2k\" (UID: \"1b651297-b930-4d64-9e35-e5632ba89357\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:49:48.328780 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.328760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b651297-b930-4d64-9e35-e5632ba89357-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-h5j2k\" (UID: \"1b651297-b930-4d64-9e35-e5632ba89357\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:49:48.350704 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.350674    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwkp\" (UniqueName: \"kubernetes.io/projected/1b651297-b930-4d64-9e35-e5632ba89357-kube-api-access-hkwkp\") pod \"kuadrant-operator-controller-manager-58fc56757c-h5j2k\" (UID: \"1b651297-b930-4d64-9e35-e5632ba89357\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:49:48.475955 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.475920    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:49:48.616799 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:48.616778    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"]
Aug 04 20:49:48.618275 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:49:48.618245    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b651297_b930_4d64_9e35_e5632ba89357.slice/crio-3898bf295aa96d9a0981c385af360e89e45e94d21a3521b510b0de3b350d5d21 WatchSource:0}: Error finding container 3898bf295aa96d9a0981c385af360e89e45e94d21a3521b510b0de3b350d5d21: Status 404 returned error can't find the container with id 3898bf295aa96d9a0981c385af360e89e45e94d21a3521b510b0de3b350d5d21
Aug 04 20:49:49.248437 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:49.248391    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k" event={"ID":"1b651297-b930-4d64-9e35-e5632ba89357","Type":"ContainerStarted","Data":"3898bf295aa96d9a0981c385af360e89e45e94d21a3521b510b0de3b350d5d21"}
Aug 04 20:49:53.270862 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:53.270820    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k" event={"ID":"1b651297-b930-4d64-9e35-e5632ba89357","Type":"ContainerStarted","Data":"afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c"}
Aug 04 20:49:53.271323 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:53.270962    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:49:53.296643 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:49:53.296589    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k" podStartSLOduration=1.22805998 podStartE2EDuration="5.296570219s" totalImagesPullingTime="4.068510239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:49:53.294098985 +0000 UTC m=+568.046204973" watchObservedRunningTime="2026-08-04 20:49:53.296570219 +0000 UTC m=+568.048676197"
Aug 04 20:50:00.429553 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:00.429517    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"]
Aug 04 20:50:00.430085 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:00.429715    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k" podUID="1b651297-b930-4d64-9e35-e5632ba89357" containerName="manager" containerID="cri-o://afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c" gracePeriod=10
Aug 04 20:50:00.431255 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:00.431227    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:50:00.678973 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:00.678950    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:50:00.742811 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:00.742781    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkwkp\" (UniqueName: \"kubernetes.io/projected/1b651297-b930-4d64-9e35-e5632ba89357-kube-api-access-hkwkp\") pod \"1b651297-b930-4d64-9e35-e5632ba89357\" (UID: \"1b651297-b930-4d64-9e35-e5632ba89357\") "
Aug 04 20:50:00.742957 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:00.742858    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b651297-b930-4d64-9e35-e5632ba89357-extensions-socket-volume\") pod \"1b651297-b930-4d64-9e35-e5632ba89357\" (UID: \"1b651297-b930-4d64-9e35-e5632ba89357\") "
Aug 04 20:50:00.743304 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:00.743279    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b651297-b930-4d64-9e35-e5632ba89357-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1b651297-b930-4d64-9e35-e5632ba89357" (UID: "1b651297-b930-4d64-9e35-e5632ba89357"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 20:50:00.744936 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:00.744909    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b651297-b930-4d64-9e35-e5632ba89357-kube-api-access-hkwkp" (OuterVolumeSpecName: "kube-api-access-hkwkp") pod "1b651297-b930-4d64-9e35-e5632ba89357" (UID: "1b651297-b930-4d64-9e35-e5632ba89357"). InnerVolumeSpecName "kube-api-access-hkwkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 20:50:00.843860 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:00.843827    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkwkp\" (UniqueName: \"kubernetes.io/projected/1b651297-b930-4d64-9e35-e5632ba89357-kube-api-access-hkwkp\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:50:00.843860 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:00.843857    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b651297-b930-4d64-9e35-e5632ba89357-extensions-socket-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 20:50:01.309333 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:01.309300    2565 generic.go:356] "Generic (PLEG): container finished" podID="1b651297-b930-4d64-9e35-e5632ba89357" containerID="afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c" exitCode=0
Aug 04 20:50:01.309516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:01.309362    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"
Aug 04 20:50:01.309516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:01.309366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k" event={"ID":"1b651297-b930-4d64-9e35-e5632ba89357","Type":"ContainerDied","Data":"afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c"}
Aug 04 20:50:01.309516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:01.309408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k" event={"ID":"1b651297-b930-4d64-9e35-e5632ba89357","Type":"ContainerDied","Data":"3898bf295aa96d9a0981c385af360e89e45e94d21a3521b510b0de3b350d5d21"}
Aug 04 20:50:01.309516 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:01.309431    2565 scope.go:117] "RemoveContainer" containerID="afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c"
Aug 04 20:50:01.318173 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:01.318154    2565 scope.go:117] "RemoveContainer" containerID="afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c"
Aug 04 20:50:01.318414 ip-10-0-140-103 kubenswrapper[2565]: E0804 20:50:01.318394    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c\": container with ID starting with afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c not found: ID does not exist" containerID="afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c"
Aug 04 20:50:01.318479 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:01.318422    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c"} err="failed to get container status \"afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c\": rpc error: code = NotFound desc = could not find container \"afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c\": container with ID starting with afb7ee6b77ecd473b42b4e04ddb09158fc9569f3643ab4d6bbe1ef892c9b100c not found: ID does not exist"
Aug 04 20:50:01.339313 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:01.339284    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"]
Aug 04 20:50:01.342692 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:01.342670    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h5j2k"]
Aug 04 20:50:01.760485 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:01.760453    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b651297-b930-4d64-9e35-e5632ba89357" path="/var/lib/kubelet/pods/1b651297-b930-4d64-9e35-e5632ba89357/volumes"
Aug 04 20:50:19.012636 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.012602    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:50:19.013131 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.012935    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b651297-b930-4d64-9e35-e5632ba89357" containerName="manager"
Aug 04 20:50:19.013131 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.012945    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b651297-b930-4d64-9e35-e5632ba89357" containerName="manager"
Aug 04 20:50:19.013131 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.013016    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b651297-b930-4d64-9e35-e5632ba89357" containerName="manager"
Aug 04 20:50:19.211714 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.211687    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:50:19.211714 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.211719    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:50:19.211893 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.211799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"
Aug 04 20:50:19.214005 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.213987    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Aug 04 20:50:19.408248 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.408170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqbb\" (UniqueName: \"kubernetes.io/projected/dcb2f898-677d-49cc-9e51-f1b07f36fe93-kube-api-access-6zqbb\") pod \"limitador-limitador-85d9db4d5d-pf7gc\" (UID: \"dcb2f898-677d-49cc-9e51-f1b07f36fe93\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"
Aug 04 20:50:19.408248 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.408225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcb2f898-677d-49cc-9e51-f1b07f36fe93-config-file\") pod \"limitador-limitador-85d9db4d5d-pf7gc\" (UID: \"dcb2f898-677d-49cc-9e51-f1b07f36fe93\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"
Aug 04 20:50:19.509106 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.509075    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqbb\" (UniqueName: \"kubernetes.io/projected/dcb2f898-677d-49cc-9e51-f1b07f36fe93-kube-api-access-6zqbb\") pod \"limitador-limitador-85d9db4d5d-pf7gc\" (UID: \"dcb2f898-677d-49cc-9e51-f1b07f36fe93\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"
Aug 04 20:50:19.509301 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.509125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcb2f898-677d-49cc-9e51-f1b07f36fe93-config-file\") pod \"limitador-limitador-85d9db4d5d-pf7gc\" (UID: \"dcb2f898-677d-49cc-9e51-f1b07f36fe93\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"
Aug 04 20:50:19.509802 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.509778    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcb2f898-677d-49cc-9e51-f1b07f36fe93-config-file\") pod \"limitador-limitador-85d9db4d5d-pf7gc\" (UID: \"dcb2f898-677d-49cc-9e51-f1b07f36fe93\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"
Aug 04 20:50:19.517278 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.517255    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqbb\" (UniqueName: \"kubernetes.io/projected/dcb2f898-677d-49cc-9e51-f1b07f36fe93-kube-api-access-6zqbb\") pod \"limitador-limitador-85d9db4d5d-pf7gc\" (UID: \"dcb2f898-677d-49cc-9e51-f1b07f36fe93\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"
Aug 04 20:50:19.522042 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.522023    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"
Aug 04 20:50:19.651790 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:19.651760    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:50:19.654578 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:50:19.654551    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb2f898_677d_49cc_9e51_f1b07f36fe93.slice/crio-66c028895117f961b3d9a4bbaa57de51efc66e91e6c0bd3933d68f7c00ca89bc WatchSource:0}: Error finding container 66c028895117f961b3d9a4bbaa57de51efc66e91e6c0bd3933d68f7c00ca89bc: Status 404 returned error can't find the container with id 66c028895117f961b3d9a4bbaa57de51efc66e91e6c0bd3933d68f7c00ca89bc
Aug 04 20:50:20.399123 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:20.399082    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc" event={"ID":"dcb2f898-677d-49cc-9e51-f1b07f36fe93","Type":"ContainerStarted","Data":"66c028895117f961b3d9a4bbaa57de51efc66e91e6c0bd3933d68f7c00ca89bc"}
Aug 04 20:50:21.404970 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:21.404936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc" event={"ID":"dcb2f898-677d-49cc-9e51-f1b07f36fe93","Type":"ContainerStarted","Data":"e0f5872d69996f135ca8faa03bd04af8c38504c809a3eb17b010c45ec51fd833"}
Aug 04 20:50:21.405365 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:21.404998    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"
Aug 04 20:50:21.424182 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:21.424125    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc" podStartSLOduration=2.408500084 podStartE2EDuration="3.424110172s" totalImagesPullingTime="1.015610088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:50:21.422504956 +0000 UTC m=+596.174610934" watchObservedRunningTime="2026-08-04 20:50:21.424110172 +0000 UTC m=+596.176216149"
Aug 04 20:50:25.700664 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:25.700637    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 20:50:25.708004 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:25.707980    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 20:50:32.411026 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:50:32.410996    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"
Aug 04 20:51:33.118293 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.118256    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-fxs52"]
Aug 04 20:51:33.122012 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.121992    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52"
Aug 04 20:51:33.124604 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.124580    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Aug 04 20:51:33.124715 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.124651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-52nh4\""
Aug 04 20:51:33.134172 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.134147    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-fxs52"]
Aug 04 20:51:33.218440 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.218411    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:51:33.257702 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.257670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkv5\" (UniqueName: \"kubernetes.io/projected/dafea131-0f06-41ca-92ff-3bc9c21e2471-kube-api-access-6rkv5\") pod \"payload-processing-5cc6cd444b-fxs52\" (UID: \"dafea131-0f06-41ca-92ff-3bc9c21e2471\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52"
Aug 04 20:51:33.257865 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.257734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafea131-0f06-41ca-92ff-3bc9c21e2471-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-fxs52\" (UID: \"dafea131-0f06-41ca-92ff-3bc9c21e2471\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52"
Aug 04 20:51:33.358639 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.358606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkv5\" (UniqueName: \"kubernetes.io/projected/dafea131-0f06-41ca-92ff-3bc9c21e2471-kube-api-access-6rkv5\") pod \"payload-processing-5cc6cd444b-fxs52\" (UID: \"dafea131-0f06-41ca-92ff-3bc9c21e2471\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52"
Aug 04 20:51:33.358806 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.358779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafea131-0f06-41ca-92ff-3bc9c21e2471-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-fxs52\" (UID: \"dafea131-0f06-41ca-92ff-3bc9c21e2471\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52"
Aug 04 20:51:33.359480 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.359463    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafea131-0f06-41ca-92ff-3bc9c21e2471-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-fxs52\" (UID: \"dafea131-0f06-41ca-92ff-3bc9c21e2471\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52"
Aug 04 20:51:33.367589 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.367570    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkv5\" (UniqueName: \"kubernetes.io/projected/dafea131-0f06-41ca-92ff-3bc9c21e2471-kube-api-access-6rkv5\") pod \"payload-processing-5cc6cd444b-fxs52\" (UID: \"dafea131-0f06-41ca-92ff-3bc9c21e2471\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52"
Aug 04 20:51:33.432510 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.432436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52"
Aug 04 20:51:33.773034 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:33.773014    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-fxs52"]
Aug 04 20:51:33.775320 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:51:33.775294    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafea131_0f06_41ca_92ff_3bc9c21e2471.slice/crio-38c39316bee72049beacc37ce71af540de1e67573ed32a51f952d98370d1f699 WatchSource:0}: Error finding container 38c39316bee72049beacc37ce71af540de1e67573ed32a51f952d98370d1f699: Status 404 returned error can't find the container with id 38c39316bee72049beacc37ce71af540de1e67573ed32a51f952d98370d1f699
Aug 04 20:51:34.756896 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:34.756855    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52" event={"ID":"dafea131-0f06-41ca-92ff-3bc9c21e2471","Type":"ContainerStarted","Data":"38c39316bee72049beacc37ce71af540de1e67573ed32a51f952d98370d1f699"}
Aug 04 20:51:35.768093 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:35.768054    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52" event={"ID":"dafea131-0f06-41ca-92ff-3bc9c21e2471","Type":"ContainerStarted","Data":"fa31036c7de3a76e77af60d5beaace65800710a90fbdacda8d39b66d5175d26b"}
Aug 04 20:51:35.768488 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:35.768126    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52"
Aug 04 20:51:35.785885 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:35.785840    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52" podStartSLOduration=1.34722675 podStartE2EDuration="2.785826719s" totalImagesPullingTime="1.438599969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:51:35.784654419 +0000 UTC m=+670.536760396" watchObservedRunningTime="2026-08-04 20:51:35.785826719 +0000 UTC m=+670.537932674"
Aug 04 20:51:45.851165 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:45.851130    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:51:46.776086 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:46.776058    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-fxs52"
Aug 04 20:51:56.165448 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:51:56.165416    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:52:07.652683 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:07.652642    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:52:12.450989 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.450950    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"]
Aug 04 20:52:12.456445 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.456424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.460028 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.460000    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Aug 04 20:52:12.460792 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.460767    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Aug 04 20:52:12.461104 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.461086    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Aug 04 20:52:12.461376 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.461356    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kftb7\""
Aug 04 20:52:12.467421 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.467387    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"]
Aug 04 20:52:12.504330 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.504307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.504447 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.504363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.504447 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.504388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.504447 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.504441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.504553 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.504490    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khdh\" (UniqueName: \"kubernetes.io/projected/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-kube-api-access-6khdh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.504553 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.504542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.605213 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.605187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.605369 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.605234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.605369 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.605253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.605476 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.605384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.605531 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.605480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.605531 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.605516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khdh\" (UniqueName: \"kubernetes.io/projected/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-kube-api-access-6khdh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.605635 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.605583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.605697 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.605658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.605792 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.605772    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.607495 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.607475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.607776 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.607759    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.614191 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.614168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khdh\" (UniqueName: \"kubernetes.io/projected/e3a6b9be-0cf5-456d-824c-9a7b050a38bd-kube-api-access-6khdh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw\" (UID: \"e3a6b9be-0cf5-456d-824c-9a7b050a38bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.774537 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.774508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:12.906120 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.906092    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"]
Aug 04 20:52:12.908307 ip-10-0-140-103 kubenswrapper[2565]: W0804 20:52:12.908276    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a6b9be_0cf5_456d_824c_9a7b050a38bd.slice/crio-29cd3d199cd3cce23ec2a0927dea93ec4d467197f1e1a5caec98e4d43384ac73 WatchSource:0}: Error finding container 29cd3d199cd3cce23ec2a0927dea93ec4d467197f1e1a5caec98e4d43384ac73: Status 404 returned error can't find the container with id 29cd3d199cd3cce23ec2a0927dea93ec4d467197f1e1a5caec98e4d43384ac73
Aug 04 20:52:12.940578 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:12.940551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw" event={"ID":"e3a6b9be-0cf5-456d-824c-9a7b050a38bd","Type":"ContainerStarted","Data":"29cd3d199cd3cce23ec2a0927dea93ec4d467197f1e1a5caec98e4d43384ac73"}
Aug 04 20:52:13.761076 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:13.761037    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:52:14.951907 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:14.951870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw" event={"ID":"e3a6b9be-0cf5-456d-824c-9a7b050a38bd","Type":"ContainerStarted","Data":"3b41d835b967baeaf3e82272bf673ff273df10e69a73fbc23ec91baf4c9b8d41"}
Aug 04 20:52:14.972100 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:14.972051    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw" podStartSLOduration=1.875786317 podStartE2EDuration="2.972035456s" totalImagesPullingTime="1.096249139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:52:14.970692777 +0000 UTC m=+709.722798755" watchObservedRunningTime="2026-08-04 20:52:14.972035456 +0000 UTC m=+709.724141432"
Aug 04 20:52:22.775485 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:22.775400    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:22.776040 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:22.775497    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:22.787868 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:22.787836    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:22.999855 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:22.999819    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw"
Aug 04 20:52:25.856120 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:25.856086    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:52:32.665242 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:52:32.665198    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:53:50.355155 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:53:50.355075    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:53:59.852033 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:53:59.851998    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:54:11.655886 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:54:11.655844    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:55:25.741494 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:55:25.741422    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 20:55:25.746496 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:55:25.746473    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 20:55:52.252690 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:55:52.252653    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:56:01.058060 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:56:01.058025    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:57:01.470740 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:57:01.470705    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:57:08.354100 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:57:08.354059    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:57:23.847646 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:57:23.847611    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:57:30.543602 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:57:30.543569    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:57:41.754321 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:57:41.754282    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:57:46.746816 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:57:46.746781    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:58:00.051580 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:58:00.051545    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:58:05.151037 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:58:05.150999    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:58:13.340410 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:58:13.340375    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:58:19.246066 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:58:19.245983    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:58:32.055270 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:58:32.055223    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:58:42.746431 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:58:42.746398    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:59:06.050354 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:59:06.050315    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:59:16.464470 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:59:16.464435    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:59:22.765799 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:59:22.765765    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:59:29.457997 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:59:29.457963    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:59:37.277471 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:59:37.277435    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:59:42.154279 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:59:42.154243    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:59:47.661071 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:59:47.661028    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 20:59:55.054313 ip-10-0-140-103 kubenswrapper[2565]: I0804 20:59:55.054232    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:00:00.157428 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:00.157392    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg"]
Aug 04 21:00:00.160801 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:00.160784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg"
Aug 04 21:00:00.163727 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:00.163706    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-86svs\""
Aug 04 21:00:00.170104 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:00.170086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg"
Aug 04 21:00:00.183389 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:00.183361    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg"]
Aug 04 21:00:00.302330 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:00.302303    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg"]
Aug 04 21:00:00.305044 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:00:00.305019    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41e0a8e_d0c7_4c0e_af6f_49eb25f2d474.slice/crio-04d90c7a9c9ea3dfffb22e50fb49cfbfe0f7be6a7dcd72d4a69ee244fe8bcccb WatchSource:0}: Error finding container 04d90c7a9c9ea3dfffb22e50fb49cfbfe0f7be6a7dcd72d4a69ee244fe8bcccb: Status 404 returned error can't find the container with id 04d90c7a9c9ea3dfffb22e50fb49cfbfe0f7be6a7dcd72d4a69ee244fe8bcccb
Aug 04 21:00:00.306984 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:00.306966    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 21:00:01.128585 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:01.128549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg" event={"ID":"b41e0a8e-d0c7-4c0e-af6f-49eb25f2d474","Type":"ContainerStarted","Data":"04d90c7a9c9ea3dfffb22e50fb49cfbfe0f7be6a7dcd72d4a69ee244fe8bcccb"}
Aug 04 21:00:03.141378 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:03.141282    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg" event={"ID":"b41e0a8e-d0c7-4c0e-af6f-49eb25f2d474","Type":"ContainerStarted","Data":"d570b935fe4601ca51e0c73ddb1d2137f7c367a456522ae287786ae4372bf51d"}
Aug 04 21:00:03.159262 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:03.159215    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg" podStartSLOduration=0.875863009 podStartE2EDuration="3.159200147s" totalImagesPullingTime="2.283337138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:00:03.158594257 +0000 UTC m=+1177.910700234" watchObservedRunningTime="2026-08-04 21:00:03.159200147 +0000 UTC m=+1177.911306124"
Aug 04 21:00:04.148156 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:04.148109    2565 generic.go:356] "Generic (PLEG): container finished" podID="b41e0a8e-d0c7-4c0e-af6f-49eb25f2d474" containerID="d570b935fe4601ca51e0c73ddb1d2137f7c367a456522ae287786ae4372bf51d" exitCode=0
Aug 04 21:00:04.148574 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:04.148196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg" event={"ID":"b41e0a8e-d0c7-4c0e-af6f-49eb25f2d474","Type":"ContainerDied","Data":"d570b935fe4601ca51e0c73ddb1d2137f7c367a456522ae287786ae4372bf51d"}
Aug 04 21:00:05.278305 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:05.278278    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg"
Aug 04 21:00:06.158520 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:06.158432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg" event={"ID":"b41e0a8e-d0c7-4c0e-af6f-49eb25f2d474","Type":"ContainerDied","Data":"04d90c7a9c9ea3dfffb22e50fb49cfbfe0f7be6a7dcd72d4a69ee244fe8bcccb"}
Aug 04 21:00:06.158520 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:06.158450    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764620-55wcg"
Aug 04 21:00:06.158520 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:06.158477    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d90c7a9c9ea3dfffb22e50fb49cfbfe0f7be6a7dcd72d4a69ee244fe8bcccb"
Aug 04 21:00:09.763385 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:09.763353    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:00:14.762482 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:14.762446    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:00:25.158038 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:25.157999    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:00:25.774932 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:25.774890    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:00:25.776276 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:25.776253    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:00:30.067589 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:30.067554    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:00:40.145986 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:40.145950    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:00:46.147469 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:46.147435    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:00:51.195317 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:00:51.195278    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:01:01.354944 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:01:01.354906    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:03:17.152529 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:03:17.152494    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:03:22.145677 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:03:22.145633    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:03:45.946172 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:03:45.946138    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:04:22.459249 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:04:22.459211    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:04:33.254228 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:04:33.254191    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:04:38.342228 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:04:38.342192    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:04:43.243088 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:04:43.243049    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:04:53.053814 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:04:53.053782    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:04:59.453406 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:04:59.453370    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:05:08.452730 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:05:08.452695    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:05:13.653190 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:05:13.653153    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:05:23.753038 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:05:23.753003    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:05:25.805096 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:05:25.805069    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:05:25.806041 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:05:25.806022    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:05:30.254250 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:05:30.254216    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:05:40.864365 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:05:40.864309    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:06:04.138531 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:04.138495    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:06:11.043128 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:11.043092    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:06:33.352551 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.352516    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"]
Aug 04 21:06:33.352925 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.352909    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b41e0a8e-d0c7-4c0e-af6f-49eb25f2d474" containerName="cleanup"
Aug 04 21:06:33.352971 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.352927    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b41e0a8e-d0c7-4c0e-af6f-49eb25f2d474" containerName="cleanup"
Aug 04 21:06:33.353018 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.352999    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b41e0a8e-d0c7-4c0e-af6f-49eb25f2d474" containerName="cleanup"
Aug 04 21:06:33.356280 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.356259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.360722 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.359362    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5bd3fe37-kserve-self-signed-certs\""
Aug 04 21:06:33.374439 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.374415    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"]
Aug 04 21:06:33.466208 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.466176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-home\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.466208 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.466206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tmp-dir\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.466416 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.466229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv9s7\" (UniqueName: \"kubernetes.io/projected/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-kube-api-access-cv9s7\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.466416 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.466330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-model-cache\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.466486 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.466416    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-dshm\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.466486 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.466440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tls-certs\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.567621 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.567588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-model-cache\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.567793 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.567633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-dshm\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.567793 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.567654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tls-certs\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.567793 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.567705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-home\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.567793 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.567719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tmp-dir\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.567793 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.567746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9s7\" (UniqueName: \"kubernetes.io/projected/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-kube-api-access-cv9s7\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.568165 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.568121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-model-cache\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.568165 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.568147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-home\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.568309 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.568166    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tmp-dir\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.569918 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.569899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-dshm\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.569998 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.569981    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tls-certs\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.576526 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.576508    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9s7\" (UniqueName: \"kubernetes.io/projected/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-kube-api-access-cv9s7\") pod \"e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") " pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.672748 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.672680    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:33.805381 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.805328    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"]
Aug 04 21:06:33.807675 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:06:33.807642    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc83c0a97_ce5d_4f4e_ac19_499dc6d4704f.slice/crio-efcd8a09fd853c028ee0cb1f9ab081ea743c1ad700ede18b0b121b66706b9436 WatchSource:0}: Error finding container efcd8a09fd853c028ee0cb1f9ab081ea743c1ad700ede18b0b121b66706b9436: Status 404 returned error can't find the container with id efcd8a09fd853c028ee0cb1f9ab081ea743c1ad700ede18b0b121b66706b9436
Aug 04 21:06:33.985548 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.985507    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww" event={"ID":"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f","Type":"ContainerStarted","Data":"9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935"}
Aug 04 21:06:33.985738 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:33.985558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww" event={"ID":"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f","Type":"ContainerStarted","Data":"efcd8a09fd853c028ee0cb1f9ab081ea743c1ad700ede18b0b121b66706b9436"}
Aug 04 21:06:34.005800 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:34.005736    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww" podStartSLOduration=1.005712746 podStartE2EDuration="1.005712746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:06:34.004401339 +0000 UTC m=+1568.756507312" watchObservedRunningTime="2026-08-04 21:06:34.005712746 +0000 UTC m=+1568.757818724"
Aug 04 21:06:35.660589 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:35.660556    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:06:43.673388 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:43.673278    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:43.673388 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:43.673369    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:43.685691 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:43.685670    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:44.044370 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:44.044327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:06:50.044452 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.044418    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"]
Aug 04 21:06:50.048377 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.048337    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.050711 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.050693    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5bd3fe37-kserve-self-signed-certs\""
Aug 04 21:06:50.058560 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.058535    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"]
Aug 04 21:06:50.122096 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.122065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcps6\" (UniqueName: \"kubernetes.io/projected/95346d05-12bc-46db-8fdc-9a971ebd774d-kube-api-access-jcps6\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.122229 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.122123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-dshm\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.122229 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.122157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-tmp-dir\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.122229 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.122179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95346d05-12bc-46db-8fdc-9a971ebd774d-tls-certs\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.122229 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.122209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-home\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.122403 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.122248    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-model-cache\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.223007 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.222980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-dshm\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.223007 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.223011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-tmp-dir\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.223258 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.223041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95346d05-12bc-46db-8fdc-9a971ebd774d-tls-certs\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.223258 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.223078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-home\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.223258 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.223105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-model-cache\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.223258 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.223197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcps6\" (UniqueName: \"kubernetes.io/projected/95346d05-12bc-46db-8fdc-9a971ebd774d-kube-api-access-jcps6\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.223556 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.223519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-tmp-dir\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.223639 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.223545    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-home\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.223639 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.223596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-model-cache\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.225302 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.225281    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-dshm\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.225485 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.225469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95346d05-12bc-46db-8fdc-9a971ebd774d-tls-certs\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.232507 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.232486    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcps6\" (UniqueName: \"kubernetes.io/projected/95346d05-12bc-46db-8fdc-9a971ebd774d-kube-api-access-jcps6\") pod \"e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") " pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.358874 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.358780    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:06:50.496729 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:50.496703    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"]
Aug 04 21:06:50.499121 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:06:50.499089    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95346d05_12bc_46db_8fdc_9a971ebd774d.slice/crio-8ba64845356dd4452406cff68bea673bcd8da5d54baa484f858f7af69f0f8171 WatchSource:0}: Error finding container 8ba64845356dd4452406cff68bea673bcd8da5d54baa484f858f7af69f0f8171: Status 404 returned error can't find the container with id 8ba64845356dd4452406cff68bea673bcd8da5d54baa484f858f7af69f0f8171
Aug 04 21:06:51.066145 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:51.066108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh" event={"ID":"95346d05-12bc-46db-8fdc-9a971ebd774d","Type":"ContainerStarted","Data":"18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102"}
Aug 04 21:06:51.066145 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:51.066151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh" event={"ID":"95346d05-12bc-46db-8fdc-9a971ebd774d","Type":"ContainerStarted","Data":"8ba64845356dd4452406cff68bea673bcd8da5d54baa484f858f7af69f0f8171"}
Aug 04 21:06:51.083367 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:51.083305    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh" podStartSLOduration=1.083290337 podStartE2EDuration="1.083290337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:06:51.082671591 +0000 UTC m=+1585.834777568" watchObservedRunningTime="2026-08-04 21:06:51.083290337 +0000 UTC m=+1585.835396313"
Aug 04 21:06:52.747050 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:06:52.747014    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:07:00.359437 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:00.359400    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:07:00.359812 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:00.359449    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:07:00.372162 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:00.372137    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:07:01.129319 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:01.129285    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:07:12.323531 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:12.323498    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"]
Aug 04 21:07:13.169358 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:13.169310    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"]
Aug 04 21:07:15.583149 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:15.583116    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:07:25.761998 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:25.761929    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:07:27.342817 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.342777    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww" podUID="c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" containerName="main" containerID="cri-o://9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935" gracePeriod=45
Aug 04 21:07:27.589387 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.589360    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:07:27.652836 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.652772    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-dshm\") pod \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") "
Aug 04 21:07:27.652836 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.652821    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tmp-dir\") pod \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") "
Aug 04 21:07:27.653031 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.652863    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-home\") pod \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") "
Aug 04 21:07:27.653031 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.652904    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tls-certs\") pod \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") "
Aug 04 21:07:27.653031 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.652927    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv9s7\" (UniqueName: \"kubernetes.io/projected/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-kube-api-access-cv9s7\") pod \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") "
Aug 04 21:07:27.653182 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.653080    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-model-cache\") pod \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\" (UID: \"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f\") "
Aug 04 21:07:27.653182 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.653101    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-home" (OuterVolumeSpecName: "home") pod "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" (UID: "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:07:27.653182 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.653134    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" (UID: "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:07:27.653334 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.653308    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" (UID: "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:07:27.653465 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.653441    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:27.653465 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.653464    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:27.653614 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.653480    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:27.655024 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.654996    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-dshm" (OuterVolumeSpecName: "dshm") pod "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" (UID: "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:07:27.655024 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.655007    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" (UID: "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 21:07:27.655147 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.655067    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-kube-api-access-cv9s7" (OuterVolumeSpecName: "kube-api-access-cv9s7") pod "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" (UID: "c83c0a97-ce5d-4f4e-ac19-499dc6d4704f"). InnerVolumeSpecName "kube-api-access-cv9s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:07:27.754983 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.754952    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:27.755110 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.755006    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv9s7\" (UniqueName: \"kubernetes.io/projected/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-kube-api-access-cv9s7\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:27.755110 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:27.755033    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:28.188373 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.188307    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh" podUID="95346d05-12bc-46db-8fdc-9a971ebd774d" containerName="main" containerID="cri-o://18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102" gracePeriod=45
Aug 04 21:07:28.254043 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.254014    2565 generic.go:356] "Generic (PLEG): container finished" podID="c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" containerID="9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935" exitCode=0
Aug 04 21:07:28.254187 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.254058    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"
Aug 04 21:07:28.254187 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.254105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww" event={"ID":"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f","Type":"ContainerDied","Data":"9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935"}
Aug 04 21:07:28.254187 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.254152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww" event={"ID":"c83c0a97-ce5d-4f4e-ac19-499dc6d4704f","Type":"ContainerDied","Data":"efcd8a09fd853c028ee0cb1f9ab081ea743c1ad700ede18b0b121b66706b9436"}
Aug 04 21:07:28.254187 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.254177    2565 scope.go:117] "RemoveContainer" containerID="9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935"
Aug 04 21:07:28.270467 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.270444    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"]
Aug 04 21:07:28.274352 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.274326    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5bd3fe37-kserve-8648d5d84c-mlsww"]
Aug 04 21:07:28.298536 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.298515    2565 scope.go:117] "RemoveContainer" containerID="9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935"
Aug 04 21:07:28.298820 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:07:28.298800    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935\": container with ID starting with 9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935 not found: ID does not exist" containerID="9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935"
Aug 04 21:07:28.298901 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.298827    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935"} err="failed to get container status \"9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935\": rpc error: code = NotFound desc = could not find container \"9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935\": container with ID starting with 9e78c171a28191fa445c88b90c0d72635ec69f98854314d1b2bc4b9b46efd935 not found: ID does not exist"
Aug 04 21:07:28.438366 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.438328    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:07:28.562542 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.562508    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-dshm\") pod \"95346d05-12bc-46db-8fdc-9a971ebd774d\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") "
Aug 04 21:07:28.562542 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.562543    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-model-cache\") pod \"95346d05-12bc-46db-8fdc-9a971ebd774d\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") "
Aug 04 21:07:28.562785 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.562568    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-tmp-dir\") pod \"95346d05-12bc-46db-8fdc-9a971ebd774d\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") "
Aug 04 21:07:28.562785 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.562599    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcps6\" (UniqueName: \"kubernetes.io/projected/95346d05-12bc-46db-8fdc-9a971ebd774d-kube-api-access-jcps6\") pod \"95346d05-12bc-46db-8fdc-9a971ebd774d\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") "
Aug 04 21:07:28.562785 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.562641    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95346d05-12bc-46db-8fdc-9a971ebd774d-tls-certs\") pod \"95346d05-12bc-46db-8fdc-9a971ebd774d\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") "
Aug 04 21:07:28.562785 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.562656    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-home\") pod \"95346d05-12bc-46db-8fdc-9a971ebd774d\" (UID: \"95346d05-12bc-46db-8fdc-9a971ebd774d\") "
Aug 04 21:07:28.562943 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.562802    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-model-cache" (OuterVolumeSpecName: "model-cache") pod "95346d05-12bc-46db-8fdc-9a971ebd774d" (UID: "95346d05-12bc-46db-8fdc-9a971ebd774d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:07:28.562943 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.562828    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95346d05-12bc-46db-8fdc-9a971ebd774d" (UID: "95346d05-12bc-46db-8fdc-9a971ebd774d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:07:28.562943 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.562879    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:28.563059 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.563026    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-home" (OuterVolumeSpecName: "home") pod "95346d05-12bc-46db-8fdc-9a971ebd774d" (UID: "95346d05-12bc-46db-8fdc-9a971ebd774d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:07:28.564709 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.564685    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-dshm" (OuterVolumeSpecName: "dshm") pod "95346d05-12bc-46db-8fdc-9a971ebd774d" (UID: "95346d05-12bc-46db-8fdc-9a971ebd774d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:07:28.564817 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.564731    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95346d05-12bc-46db-8fdc-9a971ebd774d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95346d05-12bc-46db-8fdc-9a971ebd774d" (UID: "95346d05-12bc-46db-8fdc-9a971ebd774d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 21:07:28.564817 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.564803    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95346d05-12bc-46db-8fdc-9a971ebd774d-kube-api-access-jcps6" (OuterVolumeSpecName: "kube-api-access-jcps6") pod "95346d05-12bc-46db-8fdc-9a971ebd774d" (UID: "95346d05-12bc-46db-8fdc-9a971ebd774d"). InnerVolumeSpecName "kube-api-access-jcps6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:07:28.663843 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.663821    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:28.663954 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.663860    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:28.663954 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.663872    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcps6\" (UniqueName: \"kubernetes.io/projected/95346d05-12bc-46db-8fdc-9a971ebd774d-kube-api-access-jcps6\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:28.663954 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.663880    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95346d05-12bc-46db-8fdc-9a971ebd774d-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:28.663954 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:28.663888    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95346d05-12bc-46db-8fdc-9a971ebd774d-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:07:29.261878 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:29.261846    2565 generic.go:356] "Generic (PLEG): container finished" podID="95346d05-12bc-46db-8fdc-9a971ebd774d" containerID="18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102" exitCode=0
Aug 04 21:07:29.262056 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:29.261898    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"
Aug 04 21:07:29.262056 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:29.261940    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh" event={"ID":"95346d05-12bc-46db-8fdc-9a971ebd774d","Type":"ContainerDied","Data":"18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102"}
Aug 04 21:07:29.262056 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:29.261984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh" event={"ID":"95346d05-12bc-46db-8fdc-9a971ebd774d","Type":"ContainerDied","Data":"8ba64845356dd4452406cff68bea673bcd8da5d54baa484f858f7af69f0f8171"}
Aug 04 21:07:29.262056 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:29.262002    2565 scope.go:117] "RemoveContainer" containerID="18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102"
Aug 04 21:07:29.271014 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:29.270995    2565 scope.go:117] "RemoveContainer" containerID="18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102"
Aug 04 21:07:29.271250 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:07:29.271232    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102\": container with ID starting with 18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102 not found: ID does not exist" containerID="18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102"
Aug 04 21:07:29.271297 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:29.271260    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102"} err="failed to get container status \"18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102\": rpc error: code = NotFound desc = could not find container \"18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102\": container with ID starting with 18fa840aab8bb03bec27b5dea02d26e6d7b8a6decac7e250e11c8741aee66102 not found: ID does not exist"
Aug 04 21:07:29.283780 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:29.283759    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"]
Aug 04 21:07:29.287169 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:29.287150    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5bd3fe37-kserve-5cffb69679-nf6gh"]
Aug 04 21:07:29.760879 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:29.760846    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95346d05-12bc-46db-8fdc-9a971ebd774d" path="/var/lib/kubelet/pods/95346d05-12bc-46db-8fdc-9a971ebd774d/volumes"
Aug 04 21:07:29.761248 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:29.761226    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" path="/var/lib/kubelet/pods/c83c0a97-ce5d-4f4e-ac19-499dc6d4704f/volumes"
Aug 04 21:07:30.545872 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:30.545837    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:07:37.367138 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:37.367105    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:07:46.743167 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:46.743132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:07:52.252410 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:52.252378    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:07:59.848581 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:07:59.848543    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:08:06.554578 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:08:06.554543    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:08:11.539567 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:08:11.539530    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:08:21.852060 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:08:21.852026    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:08:28.752093 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:08:28.752061    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:08:45.646174 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:08:45.646141    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:08:52.245204 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:08:52.245125    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:09:04.160261 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:09:04.160217    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:09:11.242526 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:09:11.242478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:09:17.653896 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:09:17.653858    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:09:22.552185 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:09:22.552152    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:09:29.349732 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:09:29.349698    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:09:39.046145 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:09:39.046112    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:09:45.549518 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:09:45.549488    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:09:56.055050 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:09:56.055015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:10:02.344985 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:10:02.344949    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:10:13.145396 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:10:13.145360    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:10:25.834748 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:10:25.834721    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:10:25.836391 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:10:25.836367    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:10:28.341684 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:10:28.341650    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:10:34.350089 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:10:34.350055    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:10:40.812734 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:10:40.812697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:10:52.655148 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:10:52.655109    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:11:00.745657 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:00.745623    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:11:05.961389 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:05.961358    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:11:18.415744 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.415711    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"]
Aug 04 21:11:18.416154 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.416135    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" containerName="main"
Aug 04 21:11:18.416154 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.416155    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" containerName="main"
Aug 04 21:11:18.416268 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.416164    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95346d05-12bc-46db-8fdc-9a971ebd774d" containerName="main"
Aug 04 21:11:18.416268 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.416169    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="95346d05-12bc-46db-8fdc-9a971ebd774d" containerName="main"
Aug 04 21:11:18.416268 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.416227    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83c0a97-ce5d-4f4e-ac19-499dc6d4704f" containerName="main"
Aug 04 21:11:18.416268 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.416242    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="95346d05-12bc-46db-8fdc-9a971ebd774d" containerName="main"
Aug 04 21:11:18.419406 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.419388    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:18.421916 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.421896    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5a7758b8\""
Aug 04 21:11:18.422019 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.421934    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5a7758b8-dockercfg-tttc7\""
Aug 04 21:11:18.429013 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.428986    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"]
Aug 04 21:11:18.457567 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.457536    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"]
Aug 04 21:11:18.462238 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.462219    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:18.469885 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.469859    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"]
Aug 04 21:11:18.555555 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.555521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3768b147-c140-4004-a403-daa8de0e8d02-plugins-config-volume\") pod \"payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2\" (UID: \"3768b147-c140-4004-a403-daa8de0e8d02\") " pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:18.555711 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.555568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b687ed-901c-4eba-aa9d-5dae896d2ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9\" (UID: \"74b687ed-901c-4eba-aa9d-5dae896d2ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:18.555711 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.555591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6csg4\" (UniqueName: \"kubernetes.io/projected/74b687ed-901c-4eba-aa9d-5dae896d2ded-kube-api-access-6csg4\") pod \"payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9\" (UID: \"74b687ed-901c-4eba-aa9d-5dae896d2ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:18.555711 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.555657    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpbr2\" (UniqueName: \"kubernetes.io/projected/3768b147-c140-4004-a403-daa8de0e8d02-kube-api-access-gpbr2\") pod \"payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2\" (UID: \"3768b147-c140-4004-a403-daa8de0e8d02\") " pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:18.657103 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.657070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3768b147-c140-4004-a403-daa8de0e8d02-plugins-config-volume\") pod \"payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2\" (UID: \"3768b147-c140-4004-a403-daa8de0e8d02\") " pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:18.657247 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.657114    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b687ed-901c-4eba-aa9d-5dae896d2ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9\" (UID: \"74b687ed-901c-4eba-aa9d-5dae896d2ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:18.657315 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.657296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6csg4\" (UniqueName: \"kubernetes.io/projected/74b687ed-901c-4eba-aa9d-5dae896d2ded-kube-api-access-6csg4\") pod \"payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9\" (UID: \"74b687ed-901c-4eba-aa9d-5dae896d2ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:18.657411 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.657392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpbr2\" (UniqueName: \"kubernetes.io/projected/3768b147-c140-4004-a403-daa8de0e8d02-kube-api-access-gpbr2\") pod \"payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2\" (UID: \"3768b147-c140-4004-a403-daa8de0e8d02\") " pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:18.657817 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.657796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3768b147-c140-4004-a403-daa8de0e8d02-plugins-config-volume\") pod \"payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2\" (UID: \"3768b147-c140-4004-a403-daa8de0e8d02\") " pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:18.657908 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.657891    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b687ed-901c-4eba-aa9d-5dae896d2ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9\" (UID: \"74b687ed-901c-4eba-aa9d-5dae896d2ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:18.665022 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.664993    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6csg4\" (UniqueName: \"kubernetes.io/projected/74b687ed-901c-4eba-aa9d-5dae896d2ded-kube-api-access-6csg4\") pod \"payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9\" (UID: \"74b687ed-901c-4eba-aa9d-5dae896d2ded\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:18.665022 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.665012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpbr2\" (UniqueName: \"kubernetes.io/projected/3768b147-c140-4004-a403-daa8de0e8d02-kube-api-access-gpbr2\") pod \"payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2\" (UID: \"3768b147-c140-4004-a403-daa8de0e8d02\") " pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:18.731417 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.731395    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:18.772897 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.772865    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:18.920449 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:18.920420    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"]
Aug 04 21:11:18.922415 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:11:18.922382    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b687ed_901c_4eba_aa9d_5dae896d2ded.slice/crio-c6ea689052f023188236db2b3d7ad30652a8b4331bf387c2acceaa6e8d16cdee WatchSource:0}: Error finding container c6ea689052f023188236db2b3d7ad30652a8b4331bf387c2acceaa6e8d16cdee: Status 404 returned error can't find the container with id c6ea689052f023188236db2b3d7ad30652a8b4331bf387c2acceaa6e8d16cdee
Aug 04 21:11:19.081041 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:19.081017    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"]
Aug 04 21:11:19.083051 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:11:19.083027    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3768b147_c140_4004_a403_daa8de0e8d02.slice/crio-b557b18ab2d29c50bcde15039de87e29db8ff92849ecbc5d7d1b68a453cda5e2 WatchSource:0}: Error finding container b557b18ab2d29c50bcde15039de87e29db8ff92849ecbc5d7d1b68a453cda5e2: Status 404 returned error can't find the container with id b557b18ab2d29c50bcde15039de87e29db8ff92849ecbc5d7d1b68a453cda5e2
Aug 04 21:11:19.336272 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:19.336174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2" event={"ID":"3768b147-c140-4004-a403-daa8de0e8d02","Type":"ContainerStarted","Data":"41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36"}
Aug 04 21:11:19.336272 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:19.336234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2" event={"ID":"3768b147-c140-4004-a403-daa8de0e8d02","Type":"ContainerStarted","Data":"b557b18ab2d29c50bcde15039de87e29db8ff92849ecbc5d7d1b68a453cda5e2"}
Aug 04 21:11:19.336272 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:19.336264    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:19.337921 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:19.337887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9" event={"ID":"74b687ed-901c-4eba-aa9d-5dae896d2ded","Type":"ContainerStarted","Data":"d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2"}
Aug 04 21:11:19.338060 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:19.337929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9" event={"ID":"74b687ed-901c-4eba-aa9d-5dae896d2ded","Type":"ContainerStarted","Data":"c6ea689052f023188236db2b3d7ad30652a8b4331bf387c2acceaa6e8d16cdee"}
Aug 04 21:11:19.338060 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:19.338032    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:19.354014 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:19.353958    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2" podStartSLOduration=1.353940181 podStartE2EDuration="1.353940181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:11:19.351859547 +0000 UTC m=+1854.103965524" watchObservedRunningTime="2026-08-04 21:11:19.353940181 +0000 UTC m=+1854.106046159"
Aug 04 21:11:19.367494 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:19.367454    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9" podStartSLOduration=1.367440518 podStartE2EDuration="1.367440518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:11:19.365764224 +0000 UTC m=+1854.117870198" watchObservedRunningTime="2026-08-04 21:11:19.367440518 +0000 UTC m=+1854.119546495"
Aug 04 21:11:20.646113 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:20.646079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:11:30.344763 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:30.344729    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:30.345176 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:30.344816    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:48.481143 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.481036    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"]
Aug 04 21:11:48.481631 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.481283    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2" podUID="3768b147-c140-4004-a403-daa8de0e8d02" containerName="payload-processing" containerID="cri-o://41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36" gracePeriod=30
Aug 04 21:11:48.497913 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.497886    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"]
Aug 04 21:11:48.498204 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.498102    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9" podUID="74b687ed-901c-4eba-aa9d-5dae896d2ded" containerName="payload-pre-processing" containerID="cri-o://d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2" gracePeriod=30
Aug 04 21:11:48.742413 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.742354    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:48.767864 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.767839    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:48.807430 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.807400    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b687ed-901c-4eba-aa9d-5dae896d2ded-plugins-config-volume\") pod \"74b687ed-901c-4eba-aa9d-5dae896d2ded\" (UID: \"74b687ed-901c-4eba-aa9d-5dae896d2ded\") "
Aug 04 21:11:48.807604 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.807440    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6csg4\" (UniqueName: \"kubernetes.io/projected/74b687ed-901c-4eba-aa9d-5dae896d2ded-kube-api-access-6csg4\") pod \"74b687ed-901c-4eba-aa9d-5dae896d2ded\" (UID: \"74b687ed-901c-4eba-aa9d-5dae896d2ded\") "
Aug 04 21:11:48.807604 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.807474    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpbr2\" (UniqueName: \"kubernetes.io/projected/3768b147-c140-4004-a403-daa8de0e8d02-kube-api-access-gpbr2\") pod \"3768b147-c140-4004-a403-daa8de0e8d02\" (UID: \"3768b147-c140-4004-a403-daa8de0e8d02\") "
Aug 04 21:11:48.807604 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.807528    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3768b147-c140-4004-a403-daa8de0e8d02-plugins-config-volume\") pod \"3768b147-c140-4004-a403-daa8de0e8d02\" (UID: \"3768b147-c140-4004-a403-daa8de0e8d02\") "
Aug 04 21:11:48.807916 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.807885    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b687ed-901c-4eba-aa9d-5dae896d2ded-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74b687ed-901c-4eba-aa9d-5dae896d2ded" (UID: "74b687ed-901c-4eba-aa9d-5dae896d2ded"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:11:48.808018 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.807957    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3768b147-c140-4004-a403-daa8de0e8d02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3768b147-c140-4004-a403-daa8de0e8d02" (UID: "3768b147-c140-4004-a403-daa8de0e8d02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:11:48.809783 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.809755    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3768b147-c140-4004-a403-daa8de0e8d02-kube-api-access-gpbr2" (OuterVolumeSpecName: "kube-api-access-gpbr2") pod "3768b147-c140-4004-a403-daa8de0e8d02" (UID: "3768b147-c140-4004-a403-daa8de0e8d02"). InnerVolumeSpecName "kube-api-access-gpbr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:11:48.810059 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.810038    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b687ed-901c-4eba-aa9d-5dae896d2ded-kube-api-access-6csg4" (OuterVolumeSpecName: "kube-api-access-6csg4") pod "74b687ed-901c-4eba-aa9d-5dae896d2ded" (UID: "74b687ed-901c-4eba-aa9d-5dae896d2ded"). InnerVolumeSpecName "kube-api-access-6csg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:11:48.908584 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.908551    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3768b147-c140-4004-a403-daa8de0e8d02-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:11:48.908584 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.908578    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b687ed-901c-4eba-aa9d-5dae896d2ded-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:11:48.908584 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.908588    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6csg4\" (UniqueName: \"kubernetes.io/projected/74b687ed-901c-4eba-aa9d-5dae896d2ded-kube-api-access-6csg4\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:11:48.908810 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:48.908596    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpbr2\" (UniqueName: \"kubernetes.io/projected/3768b147-c140-4004-a403-daa8de0e8d02-kube-api-access-gpbr2\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:11:49.497194 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.497155    2565 generic.go:356] "Generic (PLEG): container finished" podID="74b687ed-901c-4eba-aa9d-5dae896d2ded" containerID="d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2" exitCode=0
Aug 04 21:11:49.497671 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.497213    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"
Aug 04 21:11:49.497671 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.497240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9" event={"ID":"74b687ed-901c-4eba-aa9d-5dae896d2ded","Type":"ContainerDied","Data":"d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2"}
Aug 04 21:11:49.497671 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.497334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9" event={"ID":"74b687ed-901c-4eba-aa9d-5dae896d2ded","Type":"ContainerDied","Data":"c6ea689052f023188236db2b3d7ad30652a8b4331bf387c2acceaa6e8d16cdee"}
Aug 04 21:11:49.497671 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.497369    2565 scope.go:117] "RemoveContainer" containerID="d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2"
Aug 04 21:11:49.498764 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.498721    2565 generic.go:356] "Generic (PLEG): container finished" podID="3768b147-c140-4004-a403-daa8de0e8d02" containerID="41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36" exitCode=0
Aug 04 21:11:49.498868 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.498780    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"
Aug 04 21:11:49.498868 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.498782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2" event={"ID":"3768b147-c140-4004-a403-daa8de0e8d02","Type":"ContainerDied","Data":"41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36"}
Aug 04 21:11:49.498868 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.498848    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2" event={"ID":"3768b147-c140-4004-a403-daa8de0e8d02","Type":"ContainerDied","Data":"b557b18ab2d29c50bcde15039de87e29db8ff92849ecbc5d7d1b68a453cda5e2"}
Aug 04 21:11:49.506878 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.506858    2565 scope.go:117] "RemoveContainer" containerID="d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2"
Aug 04 21:11:49.507122 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:11:49.507098    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2\": container with ID starting with d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2 not found: ID does not exist" containerID="d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2"
Aug 04 21:11:49.507220 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.507128    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2"} err="failed to get container status \"d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2\": rpc error: code = NotFound desc = could not find container \"d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2\": container with ID starting with d3afd033973aa4d001581b56fd9fe44a0923f88bdaf8bff8ecc6034ab8162aa2 not found: ID does not exist"
Aug 04 21:11:49.507220 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.507143    2565 scope.go:117] "RemoveContainer" containerID="41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36"
Aug 04 21:11:49.515123 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.515108    2565 scope.go:117] "RemoveContainer" containerID="41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36"
Aug 04 21:11:49.515396 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:11:49.515375    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36\": container with ID starting with 41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36 not found: ID does not exist" containerID="41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36"
Aug 04 21:11:49.515490 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.515403    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36"} err="failed to get container status \"41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36\": rpc error: code = NotFound desc = could not find container \"41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36\": container with ID starting with 41c53a8ec70056fab904ece98eac3c83ef098053c89a340d1f3882dabfce8b36 not found: ID does not exist"
Aug 04 21:11:49.525976 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.525958    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"]
Aug 04 21:11:49.529884 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.529859    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a7758b8-899594668-4pwz9"]
Aug 04 21:11:49.542158 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.542137    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"]
Aug 04 21:11:49.547374 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.547350    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5a7758b8-5c5df4f458-85nb2"]
Aug 04 21:11:49.761389 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.761305    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3768b147-c140-4004-a403-daa8de0e8d02" path="/var/lib/kubelet/pods/3768b147-c140-4004-a403-daa8de0e8d02/volumes"
Aug 04 21:11:49.761681 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:49.761667    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74b687ed-901c-4eba-aa9d-5dae896d2ded" path="/var/lib/kubelet/pods/74b687ed-901c-4eba-aa9d-5dae896d2ded/volumes"
Aug 04 21:11:51.762321 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:51.762292    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:11:53.989482 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:53.989448    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"]
Aug 04 21:11:53.989903 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:53.989817    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b687ed-901c-4eba-aa9d-5dae896d2ded" containerName="payload-pre-processing"
Aug 04 21:11:53.989903 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:53.989828    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b687ed-901c-4eba-aa9d-5dae896d2ded" containerName="payload-pre-processing"
Aug 04 21:11:53.989903 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:53.989835    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3768b147-c140-4004-a403-daa8de0e8d02" containerName="payload-processing"
Aug 04 21:11:53.989903 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:53.989840    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3768b147-c140-4004-a403-daa8de0e8d02" containerName="payload-processing"
Aug 04 21:11:53.989903 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:53.989896    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3768b147-c140-4004-a403-daa8de0e8d02" containerName="payload-processing"
Aug 04 21:11:53.989903 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:53.989907    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b687ed-901c-4eba-aa9d-5dae896d2ded" containerName="payload-pre-processing"
Aug 04 21:11:53.994770 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:53.994749    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:11:54.004072 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.004053    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ad77b99e-dockercfg-lzzdq\""
Aug 04 21:11:54.004678 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.004656    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ad77b99e\""
Aug 04 21:11:54.010685 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.010659    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"]
Aug 04 21:11:54.054053 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.054025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq8xz\" (UniqueName: \"kubernetes.io/projected/8319610a-ae22-4031-8406-08b108d78b7f-kube-api-access-rq8xz\") pod \"payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7\" (UID: \"8319610a-ae22-4031-8406-08b108d78b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:11:54.054197 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.054071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8319610a-ae22-4031-8406-08b108d78b7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7\" (UID: \"8319610a-ae22-4031-8406-08b108d78b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:11:54.092528 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.092499    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"]
Aug 04 21:11:54.096022 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.096005    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:11:54.105650 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.105628    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"]
Aug 04 21:11:54.154806 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.154777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq8xz\" (UniqueName: \"kubernetes.io/projected/8319610a-ae22-4031-8406-08b108d78b7f-kube-api-access-rq8xz\") pod \"payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7\" (UID: \"8319610a-ae22-4031-8406-08b108d78b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:11:54.154930 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.154830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwss\" (UniqueName: \"kubernetes.io/projected/d24b3a54-db31-4588-8af5-560231095f7f-kube-api-access-szwss\") pod \"payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z\" (UID: \"d24b3a54-db31-4588-8af5-560231095f7f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:11:54.154930 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.154864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8319610a-ae22-4031-8406-08b108d78b7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7\" (UID: \"8319610a-ae22-4031-8406-08b108d78b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:11:54.155059 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.154971    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24b3a54-db31-4588-8af5-560231095f7f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z\" (UID: \"d24b3a54-db31-4588-8af5-560231095f7f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:11:54.155586 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.155565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8319610a-ae22-4031-8406-08b108d78b7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7\" (UID: \"8319610a-ae22-4031-8406-08b108d78b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:11:54.162657 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.162631    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq8xz\" (UniqueName: \"kubernetes.io/projected/8319610a-ae22-4031-8406-08b108d78b7f-kube-api-access-rq8xz\") pod \"payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7\" (UID: \"8319610a-ae22-4031-8406-08b108d78b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:11:54.256436 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.256361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24b3a54-db31-4588-8af5-560231095f7f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z\" (UID: \"d24b3a54-db31-4588-8af5-560231095f7f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:11:54.256563 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.256450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwss\" (UniqueName: \"kubernetes.io/projected/d24b3a54-db31-4588-8af5-560231095f7f-kube-api-access-szwss\") pod \"payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z\" (UID: \"d24b3a54-db31-4588-8af5-560231095f7f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:11:54.256941 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.256921    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24b3a54-db31-4588-8af5-560231095f7f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z\" (UID: \"d24b3a54-db31-4588-8af5-560231095f7f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:11:54.264510 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.264488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwss\" (UniqueName: \"kubernetes.io/projected/d24b3a54-db31-4588-8af5-560231095f7f-kube-api-access-szwss\") pod \"payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z\" (UID: \"d24b3a54-db31-4588-8af5-560231095f7f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:11:54.305300 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.305275    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:11:54.407741 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.407703    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:11:54.436302 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.436274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"]
Aug 04 21:11:54.438300 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:11:54.438265    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8319610a_ae22_4031_8406_08b108d78b7f.slice/crio-4f503640b899133d9fc88d8c435ea292f2d44197624dc7c2cd6874d2f072bd2d WatchSource:0}: Error finding container 4f503640b899133d9fc88d8c435ea292f2d44197624dc7c2cd6874d2f072bd2d: Status 404 returned error can't find the container with id 4f503640b899133d9fc88d8c435ea292f2d44197624dc7c2cd6874d2f072bd2d
Aug 04 21:11:54.527517 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.527433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7" event={"ID":"8319610a-ae22-4031-8406-08b108d78b7f","Type":"ContainerStarted","Data":"934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9"}
Aug 04 21:11:54.527517 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.527498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7" event={"ID":"8319610a-ae22-4031-8406-08b108d78b7f","Type":"ContainerStarted","Data":"4f503640b899133d9fc88d8c435ea292f2d44197624dc7c2cd6874d2f072bd2d"}
Aug 04 21:11:54.527737 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.527686    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:11:54.543653 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.543582    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"]
Aug 04 21:11:54.545298 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:54.545227    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7" podStartSLOduration=1.545208291 podStartE2EDuration="1.545208291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:11:54.543812743 +0000 UTC m=+1889.295918725" watchObservedRunningTime="2026-08-04 21:11:54.545208291 +0000 UTC m=+1889.297314264"
Aug 04 21:11:54.546061 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:11:54.546038    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24b3a54_db31_4588_8af5_560231095f7f.slice/crio-5356a86477a20257cbb6d1bc49e045a4c2ec3b88baaf33b55b5965b6c2783dae WatchSource:0}: Error finding container 5356a86477a20257cbb6d1bc49e045a4c2ec3b88baaf33b55b5965b6c2783dae: Status 404 returned error can't find the container with id 5356a86477a20257cbb6d1bc49e045a4c2ec3b88baaf33b55b5965b6c2783dae
Aug 04 21:11:55.535376 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:55.535332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z" event={"ID":"d24b3a54-db31-4588-8af5-560231095f7f","Type":"ContainerStarted","Data":"7d4d5196b71a1ce367398b7205ddb39fa3bdca8329b3b43a032400b572aa374c"}
Aug 04 21:11:55.535852 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:55.535556    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:11:55.535852 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:55.535597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z" event={"ID":"d24b3a54-db31-4588-8af5-560231095f7f","Type":"ContainerStarted","Data":"5356a86477a20257cbb6d1bc49e045a4c2ec3b88baaf33b55b5965b6c2783dae"}
Aug 04 21:11:55.553395 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:55.553337    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z" podStartSLOduration=1.553324143 podStartE2EDuration="1.553324143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:11:55.551575319 +0000 UTC m=+1890.303681297" watchObservedRunningTime="2026-08-04 21:11:55.553324143 +0000 UTC m=+1890.305430120"
Aug 04 21:11:59.958639 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:11:59.958607    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:12:05.537920 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:05.537892    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:12:06.542408 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:06.542372    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:12:24.556510 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.556477    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"]
Aug 04 21:12:24.556966 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.556691    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z" podUID="d24b3a54-db31-4588-8af5-560231095f7f" containerName="payload-processing" containerID="cri-o://7d4d5196b71a1ce367398b7205ddb39fa3bdca8329b3b43a032400b572aa374c" gracePeriod=30
Aug 04 21:12:24.562294 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.562265    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"]
Aug 04 21:12:24.562553 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.562509    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7" podUID="8319610a-ae22-4031-8406-08b108d78b7f" containerName="payload-pre-processing" containerID="cri-o://934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9" gracePeriod=30
Aug 04 21:12:24.675936 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.675907    2565 generic.go:356] "Generic (PLEG): container finished" podID="d24b3a54-db31-4588-8af5-560231095f7f" containerID="7d4d5196b71a1ce367398b7205ddb39fa3bdca8329b3b43a032400b572aa374c" exitCode=0
Aug 04 21:12:24.676082 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.675984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z" event={"ID":"d24b3a54-db31-4588-8af5-560231095f7f","Type":"ContainerDied","Data":"7d4d5196b71a1ce367398b7205ddb39fa3bdca8329b3b43a032400b572aa374c"}
Aug 04 21:12:24.854928 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.854896    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:12:24.861172 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.861152    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:12:24.934120 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.934088    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8319610a-ae22-4031-8406-08b108d78b7f-plugins-config-volume\") pod \"8319610a-ae22-4031-8406-08b108d78b7f\" (UID: \"8319610a-ae22-4031-8406-08b108d78b7f\") "
Aug 04 21:12:24.934120 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.934126    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24b3a54-db31-4588-8af5-560231095f7f-plugins-config-volume\") pod \"d24b3a54-db31-4588-8af5-560231095f7f\" (UID: \"d24b3a54-db31-4588-8af5-560231095f7f\") "
Aug 04 21:12:24.934382 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.934169    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq8xz\" (UniqueName: \"kubernetes.io/projected/8319610a-ae22-4031-8406-08b108d78b7f-kube-api-access-rq8xz\") pod \"8319610a-ae22-4031-8406-08b108d78b7f\" (UID: \"8319610a-ae22-4031-8406-08b108d78b7f\") "
Aug 04 21:12:24.934382 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.934208    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szwss\" (UniqueName: \"kubernetes.io/projected/d24b3a54-db31-4588-8af5-560231095f7f-kube-api-access-szwss\") pod \"d24b3a54-db31-4588-8af5-560231095f7f\" (UID: \"d24b3a54-db31-4588-8af5-560231095f7f\") "
Aug 04 21:12:24.934607 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.934580    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8319610a-ae22-4031-8406-08b108d78b7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8319610a-ae22-4031-8406-08b108d78b7f" (UID: "8319610a-ae22-4031-8406-08b108d78b7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:12:24.934669 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.934599    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24b3a54-db31-4588-8af5-560231095f7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d24b3a54-db31-4588-8af5-560231095f7f" (UID: "d24b3a54-db31-4588-8af5-560231095f7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:12:24.936303 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.936282    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8319610a-ae22-4031-8406-08b108d78b7f-kube-api-access-rq8xz" (OuterVolumeSpecName: "kube-api-access-rq8xz") pod "8319610a-ae22-4031-8406-08b108d78b7f" (UID: "8319610a-ae22-4031-8406-08b108d78b7f"). InnerVolumeSpecName "kube-api-access-rq8xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:12:24.936369 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:24.936300    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24b3a54-db31-4588-8af5-560231095f7f-kube-api-access-szwss" (OuterVolumeSpecName: "kube-api-access-szwss") pod "d24b3a54-db31-4588-8af5-560231095f7f" (UID: "d24b3a54-db31-4588-8af5-560231095f7f"). InnerVolumeSpecName "kube-api-access-szwss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:12:25.034850 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.034812    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8319610a-ae22-4031-8406-08b108d78b7f-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:12:25.034850 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.034845    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24b3a54-db31-4588-8af5-560231095f7f-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:12:25.034850 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.034855    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq8xz\" (UniqueName: \"kubernetes.io/projected/8319610a-ae22-4031-8406-08b108d78b7f-kube-api-access-rq8xz\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:12:25.035093 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.034864    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwss\" (UniqueName: \"kubernetes.io/projected/d24b3a54-db31-4588-8af5-560231095f7f-kube-api-access-szwss\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:12:25.682678 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.682639    2565 generic.go:356] "Generic (PLEG): container finished" podID="8319610a-ae22-4031-8406-08b108d78b7f" containerID="934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9" exitCode=0
Aug 04 21:12:25.683155 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.682688    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"
Aug 04 21:12:25.683155 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.682731    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7" event={"ID":"8319610a-ae22-4031-8406-08b108d78b7f","Type":"ContainerDied","Data":"934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9"}
Aug 04 21:12:25.683155 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.682838    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7" event={"ID":"8319610a-ae22-4031-8406-08b108d78b7f","Type":"ContainerDied","Data":"4f503640b899133d9fc88d8c435ea292f2d44197624dc7c2cd6874d2f072bd2d"}
Aug 04 21:12:25.683155 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.682869    2565 scope.go:117] "RemoveContainer" containerID="934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9"
Aug 04 21:12:25.684991 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.684963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z" event={"ID":"d24b3a54-db31-4588-8af5-560231095f7f","Type":"ContainerDied","Data":"5356a86477a20257cbb6d1bc49e045a4c2ec3b88baaf33b55b5965b6c2783dae"}
Aug 04 21:12:25.684991 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.684983    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"
Aug 04 21:12:25.693826 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.693802    2565 scope.go:117] "RemoveContainer" containerID="934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9"
Aug 04 21:12:25.694270 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:12:25.694082    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9\": container with ID starting with 934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9 not found: ID does not exist" containerID="934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9"
Aug 04 21:12:25.694270 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.694108    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9"} err="failed to get container status \"934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9\": rpc error: code = NotFound desc = could not find container \"934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9\": container with ID starting with 934f9477a129f097673c5a77eb81604a2dd87e9d0b7dc9390a6844dde15b8fb9 not found: ID does not exist"
Aug 04 21:12:25.694270 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.694126    2565 scope.go:117] "RemoveContainer" containerID="7d4d5196b71a1ce367398b7205ddb39fa3bdca8329b3b43a032400b572aa374c"
Aug 04 21:12:25.714538 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.714514    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"]
Aug 04 21:12:25.731262 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.731237    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ad77b99e-569d5c67c6-58rw7"]
Aug 04 21:12:25.745296 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.745274    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"]
Aug 04 21:12:25.761453 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.761430    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8319610a-ae22-4031-8406-08b108d78b7f" path="/var/lib/kubelet/pods/8319610a-ae22-4031-8406-08b108d78b7f/volumes"
Aug 04 21:12:25.761697 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:25.761684    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ad77b99e-6d89dbf45c-72p6z"]
Aug 04 21:12:27.454444 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:27.454414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:12:27.760424 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:27.760394    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d24b3a54-db31-4588-8af5-560231095f7f" path="/var/lib/kubelet/pods/d24b3a54-db31-4588-8af5-560231095f7f/volumes"
Aug 04 21:12:29.807568 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.807536    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"]
Aug 04 21:12:29.807962 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.807895    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8319610a-ae22-4031-8406-08b108d78b7f" containerName="payload-pre-processing"
Aug 04 21:12:29.807962 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.807905    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8319610a-ae22-4031-8406-08b108d78b7f" containerName="payload-pre-processing"
Aug 04 21:12:29.807962 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.807915    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24b3a54-db31-4588-8af5-560231095f7f" containerName="payload-processing"
Aug 04 21:12:29.807962 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.807921    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24b3a54-db31-4588-8af5-560231095f7f" containerName="payload-processing"
Aug 04 21:12:29.808113 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.808009    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8319610a-ae22-4031-8406-08b108d78b7f" containerName="payload-pre-processing"
Aug 04 21:12:29.808113 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.808025    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24b3a54-db31-4588-8af5-560231095f7f" containerName="payload-processing"
Aug 04 21:12:29.811178 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.811151    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:12:29.813607 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.813579    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-213b68bd\""
Aug 04 21:12:29.813722 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.813631    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-213b68bd-dockercfg-pj5lp\""
Aug 04 21:12:29.821984 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.821960    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"]
Aug 04 21:12:29.825972 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.825951    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"]
Aug 04 21:12:29.829412 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.829395    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:12:29.839088 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.839061    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"]
Aug 04 21:12:29.873639 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.873610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chndd\" (UniqueName: \"kubernetes.io/projected/c12e1062-2e05-46aa-b265-2de977142124-kube-api-access-chndd\") pod \"payload-processing-e2e-ait-213b68bd-545c6d468-bcz68\" (UID: \"c12e1062-2e05-46aa-b265-2de977142124\") " pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:12:29.873810 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.873651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12e1062-2e05-46aa-b265-2de977142124-plugins-config-volume\") pod \"payload-processing-e2e-ait-213b68bd-545c6d468-bcz68\" (UID: \"c12e1062-2e05-46aa-b265-2de977142124\") " pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:12:29.873810 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.873738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjz9w\" (UniqueName: \"kubernetes.io/projected/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-kube-api-access-jjz9w\") pod \"payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb\" (UID: \"5ea14b9c-3f6b-4406-be3a-0daa25a3af38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:12:29.873810 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.873775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb\" (UID: \"5ea14b9c-3f6b-4406-be3a-0daa25a3af38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:12:29.974386 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.974325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chndd\" (UniqueName: \"kubernetes.io/projected/c12e1062-2e05-46aa-b265-2de977142124-kube-api-access-chndd\") pod \"payload-processing-e2e-ait-213b68bd-545c6d468-bcz68\" (UID: \"c12e1062-2e05-46aa-b265-2de977142124\") " pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:12:29.974609 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.974390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12e1062-2e05-46aa-b265-2de977142124-plugins-config-volume\") pod \"payload-processing-e2e-ait-213b68bd-545c6d468-bcz68\" (UID: \"c12e1062-2e05-46aa-b265-2de977142124\") " pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:12:29.974609 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.974452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjz9w\" (UniqueName: \"kubernetes.io/projected/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-kube-api-access-jjz9w\") pod \"payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb\" (UID: \"5ea14b9c-3f6b-4406-be3a-0daa25a3af38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:12:29.974609 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.974484    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb\" (UID: \"5ea14b9c-3f6b-4406-be3a-0daa25a3af38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:12:29.975240 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.975214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12e1062-2e05-46aa-b265-2de977142124-plugins-config-volume\") pod \"payload-processing-e2e-ait-213b68bd-545c6d468-bcz68\" (UID: \"c12e1062-2e05-46aa-b265-2de977142124\") " pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:12:29.975329 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.975214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb\" (UID: \"5ea14b9c-3f6b-4406-be3a-0daa25a3af38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:12:29.982664 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.982636    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chndd\" (UniqueName: \"kubernetes.io/projected/c12e1062-2e05-46aa-b265-2de977142124-kube-api-access-chndd\") pod \"payload-processing-e2e-ait-213b68bd-545c6d468-bcz68\" (UID: \"c12e1062-2e05-46aa-b265-2de977142124\") " pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:12:29.982664 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:29.982653    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjz9w\" (UniqueName: \"kubernetes.io/projected/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-kube-api-access-jjz9w\") pod \"payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb\" (UID: \"5ea14b9c-3f6b-4406-be3a-0daa25a3af38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:12:30.122509 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.122434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:12:30.140648 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.140620    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:12:30.273718 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.273689    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"]
Aug 04 21:12:30.275274 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:12:30.275246    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea14b9c_3f6b_4406_be3a_0daa25a3af38.slice/crio-6a8980b4f3a22416f2d93a50f6fe0ecf5588ad55191cbe69f50f3cfad1040e41 WatchSource:0}: Error finding container 6a8980b4f3a22416f2d93a50f6fe0ecf5588ad55191cbe69f50f3cfad1040e41: Status 404 returned error can't find the container with id 6a8980b4f3a22416f2d93a50f6fe0ecf5588ad55191cbe69f50f3cfad1040e41
Aug 04 21:12:30.292336 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.292302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"]
Aug 04 21:12:30.294720 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:12:30.294690    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12e1062_2e05_46aa_b265_2de977142124.slice/crio-4d1bdcfe72a1afc559eab11aa6bf104052e004b74e935a1d5a14c5d83087f0da WatchSource:0}: Error finding container 4d1bdcfe72a1afc559eab11aa6bf104052e004b74e935a1d5a14c5d83087f0da: Status 404 returned error can't find the container with id 4d1bdcfe72a1afc559eab11aa6bf104052e004b74e935a1d5a14c5d83087f0da
Aug 04 21:12:30.709213 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.709173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68" event={"ID":"c12e1062-2e05-46aa-b265-2de977142124","Type":"ContainerStarted","Data":"cfa19d1a939342c4d4e38fc9d1c29099dea729a1cf3d1690061a7bf792357733"}
Aug 04 21:12:30.709445 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.709233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68" event={"ID":"c12e1062-2e05-46aa-b265-2de977142124","Type":"ContainerStarted","Data":"4d1bdcfe72a1afc559eab11aa6bf104052e004b74e935a1d5a14c5d83087f0da"}
Aug 04 21:12:30.709445 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.709299    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:12:30.710802 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.710774    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb" event={"ID":"5ea14b9c-3f6b-4406-be3a-0daa25a3af38","Type":"ContainerStarted","Data":"fee7c7afa74e47fcaace1c1c6c5910ed9ca148641bbf34648bc00f93abeb7c51"}
Aug 04 21:12:30.710929 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.710815    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb" event={"ID":"5ea14b9c-3f6b-4406-be3a-0daa25a3af38","Type":"ContainerStarted","Data":"6a8980b4f3a22416f2d93a50f6fe0ecf5588ad55191cbe69f50f3cfad1040e41"}
Aug 04 21:12:30.710929 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.710846    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:12:30.728058 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.728012    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68" podStartSLOduration=1.7279979220000001 podStartE2EDuration="1.727997922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:12:30.726205856 +0000 UTC m=+1925.478311837" watchObservedRunningTime="2026-08-04 21:12:30.727997922 +0000 UTC m=+1925.480103896"
Aug 04 21:12:30.742735 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:30.742690    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb" podStartSLOduration=1.742678691 podStartE2EDuration="1.742678691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:12:30.74109709 +0000 UTC m=+1925.493203067" watchObservedRunningTime="2026-08-04 21:12:30.742678691 +0000 UTC m=+1925.494784668"
Aug 04 21:12:36.354323 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:36.354287    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:12:41.717406 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:41.717371    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:12:41.717794 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:41.717444    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:12:59.699002 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:59.698963    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"]
Aug 04 21:12:59.699575 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:59.699187    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68" podUID="c12e1062-2e05-46aa-b265-2de977142124" containerName="payload-processing" containerID="cri-o://cfa19d1a939342c4d4e38fc9d1c29099dea729a1cf3d1690061a7bf792357733" gracePeriod=30
Aug 04 21:12:59.706133 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:59.705762    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"]
Aug 04 21:12:59.706133 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:59.705957    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb" podUID="5ea14b9c-3f6b-4406-be3a-0daa25a3af38" containerName="payload-pre-processing" containerID="cri-o://fee7c7afa74e47fcaace1c1c6c5910ed9ca148641bbf34648bc00f93abeb7c51" gracePeriod=30
Aug 04 21:12:59.852730 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:59.852700    2565 generic.go:356] "Generic (PLEG): container finished" podID="c12e1062-2e05-46aa-b265-2de977142124" containerID="cfa19d1a939342c4d4e38fc9d1c29099dea729a1cf3d1690061a7bf792357733" exitCode=0
Aug 04 21:12:59.852939 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:59.852775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68" event={"ID":"c12e1062-2e05-46aa-b265-2de977142124","Type":"ContainerDied","Data":"cfa19d1a939342c4d4e38fc9d1c29099dea729a1cf3d1690061a7bf792357733"}
Aug 04 21:12:59.854528 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:59.854500    2565 generic.go:356] "Generic (PLEG): container finished" podID="5ea14b9c-3f6b-4406-be3a-0daa25a3af38" containerID="fee7c7afa74e47fcaace1c1c6c5910ed9ca148641bbf34648bc00f93abeb7c51" exitCode=0
Aug 04 21:12:59.854666 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:59.854539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb" event={"ID":"5ea14b9c-3f6b-4406-be3a-0daa25a3af38","Type":"ContainerDied","Data":"fee7c7afa74e47fcaace1c1c6c5910ed9ca148641bbf34648bc00f93abeb7c51"}
Aug 04 21:12:59.951159 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:59.951099    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:12:59.981779 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:12:59.981755    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:13:00.047203 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.047171    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chndd\" (UniqueName: \"kubernetes.io/projected/c12e1062-2e05-46aa-b265-2de977142124-kube-api-access-chndd\") pod \"c12e1062-2e05-46aa-b265-2de977142124\" (UID: \"c12e1062-2e05-46aa-b265-2de977142124\") "
Aug 04 21:13:00.047203 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.047206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12e1062-2e05-46aa-b265-2de977142124-plugins-config-volume\") pod \"c12e1062-2e05-46aa-b265-2de977142124\" (UID: \"c12e1062-2e05-46aa-b265-2de977142124\") "
Aug 04 21:13:00.047665 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.047639    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12e1062-2e05-46aa-b265-2de977142124-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c12e1062-2e05-46aa-b265-2de977142124" (UID: "c12e1062-2e05-46aa-b265-2de977142124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:13:00.049198 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.049179    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12e1062-2e05-46aa-b265-2de977142124-kube-api-access-chndd" (OuterVolumeSpecName: "kube-api-access-chndd") pod "c12e1062-2e05-46aa-b265-2de977142124" (UID: "c12e1062-2e05-46aa-b265-2de977142124"). InnerVolumeSpecName "kube-api-access-chndd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:13:00.148701 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.148673    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjz9w\" (UniqueName: \"kubernetes.io/projected/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-kube-api-access-jjz9w\") pod \"5ea14b9c-3f6b-4406-be3a-0daa25a3af38\" (UID: \"5ea14b9c-3f6b-4406-be3a-0daa25a3af38\") "
Aug 04 21:13:00.148879 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.148716    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-plugins-config-volume\") pod \"5ea14b9c-3f6b-4406-be3a-0daa25a3af38\" (UID: \"5ea14b9c-3f6b-4406-be3a-0daa25a3af38\") "
Aug 04 21:13:00.149008 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.148989    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chndd\" (UniqueName: \"kubernetes.io/projected/c12e1062-2e05-46aa-b265-2de977142124-kube-api-access-chndd\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:13:00.149086 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.149013    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12e1062-2e05-46aa-b265-2de977142124-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:13:00.149161 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.149141    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ea14b9c-3f6b-4406-be3a-0daa25a3af38" (UID: "5ea14b9c-3f6b-4406-be3a-0daa25a3af38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:13:00.150791 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.150768    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-kube-api-access-jjz9w" (OuterVolumeSpecName: "kube-api-access-jjz9w") pod "5ea14b9c-3f6b-4406-be3a-0daa25a3af38" (UID: "5ea14b9c-3f6b-4406-be3a-0daa25a3af38"). InnerVolumeSpecName "kube-api-access-jjz9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:13:00.249917 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.249889    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:13:00.249917 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.249913    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjz9w\" (UniqueName: \"kubernetes.io/projected/5ea14b9c-3f6b-4406-be3a-0daa25a3af38-kube-api-access-jjz9w\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:13:00.861061 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.861035    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"
Aug 04 21:13:00.861061 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.861042    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68" event={"ID":"c12e1062-2e05-46aa-b265-2de977142124","Type":"ContainerDied","Data":"4d1bdcfe72a1afc559eab11aa6bf104052e004b74e935a1d5a14c5d83087f0da"}
Aug 04 21:13:00.861550 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.861108    2565 scope.go:117] "RemoveContainer" containerID="cfa19d1a939342c4d4e38fc9d1c29099dea729a1cf3d1690061a7bf792357733"
Aug 04 21:13:00.862451 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.862438    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"
Aug 04 21:13:00.862543 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.862476    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb" event={"ID":"5ea14b9c-3f6b-4406-be3a-0daa25a3af38","Type":"ContainerDied","Data":"6a8980b4f3a22416f2d93a50f6fe0ecf5588ad55191cbe69f50f3cfad1040e41"}
Aug 04 21:13:00.870664 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.870644    2565 scope.go:117] "RemoveContainer" containerID="fee7c7afa74e47fcaace1c1c6c5910ed9ca148641bbf34648bc00f93abeb7c51"
Aug 04 21:13:00.886546 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.886524    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"]
Aug 04 21:13:00.891458 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.891439    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213b68bd-b789c9f5b-dl8gb"]
Aug 04 21:13:00.904004 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.903961    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"]
Aug 04 21:13:00.905996 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:00.905978    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213b68bd-545c6d468-bcz68"]
Aug 04 21:13:01.768928 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:01.768889    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ea14b9c-3f6b-4406-be3a-0daa25a3af38" path="/var/lib/kubelet/pods/5ea14b9c-3f6b-4406-be3a-0daa25a3af38/volumes"
Aug 04 21:13:01.769537 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:01.769509    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c12e1062-2e05-46aa-b265-2de977142124" path="/var/lib/kubelet/pods/c12e1062-2e05-46aa-b265-2de977142124/volumes"
Aug 04 21:13:02.849191 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:02.849150    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:13:06.089469 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.089435    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"]
Aug 04 21:13:06.089835 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.089791    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12e1062-2e05-46aa-b265-2de977142124" containerName="payload-processing"
Aug 04 21:13:06.089835 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.089802    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12e1062-2e05-46aa-b265-2de977142124" containerName="payload-processing"
Aug 04 21:13:06.089835 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.089827    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea14b9c-3f6b-4406-be3a-0daa25a3af38" containerName="payload-pre-processing"
Aug 04 21:13:06.089835 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.089833    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea14b9c-3f6b-4406-be3a-0daa25a3af38" containerName="payload-pre-processing"
Aug 04 21:13:06.089976 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.089886    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea14b9c-3f6b-4406-be3a-0daa25a3af38" containerName="payload-pre-processing"
Aug 04 21:13:06.089976 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.089900    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c12e1062-2e05-46aa-b265-2de977142124" containerName="payload-processing"
Aug 04 21:13:06.094454 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.094434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:06.096789 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.096769    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-043d8f1f\""
Aug 04 21:13:06.096900 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.096769    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-043d8f1f-dockercfg-ptxkh\""
Aug 04 21:13:06.102032 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.102008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbdf001-13b2-494f-9220-c8edbdc09c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78\" (UID: \"1dbdf001-13b2-494f-9220-c8edbdc09c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:06.102141 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.102119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4mln\" (UniqueName: \"kubernetes.io/projected/1dbdf001-13b2-494f-9220-c8edbdc09c2f-kube-api-access-c4mln\") pod \"payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78\" (UID: \"1dbdf001-13b2-494f-9220-c8edbdc09c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:06.102965 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.102942    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"]
Aug 04 21:13:06.106669 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.106650    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"]
Aug 04 21:13:06.106756 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.106742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:06.117631 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.117606    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"]
Aug 04 21:13:06.202633 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.202605    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lzlg\" (UniqueName: \"kubernetes.io/projected/73b09150-fdf8-4bd2-a006-661b24637975-kube-api-access-7lzlg\") pod \"payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd\" (UID: \"73b09150-fdf8-4bd2-a006-661b24637975\") " pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:06.202783 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.202641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4mln\" (UniqueName: \"kubernetes.io/projected/1dbdf001-13b2-494f-9220-c8edbdc09c2f-kube-api-access-c4mln\") pod \"payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78\" (UID: \"1dbdf001-13b2-494f-9220-c8edbdc09c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:06.202783 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.202684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbdf001-13b2-494f-9220-c8edbdc09c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78\" (UID: \"1dbdf001-13b2-494f-9220-c8edbdc09c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:06.202783 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.202712    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b09150-fdf8-4bd2-a006-661b24637975-plugins-config-volume\") pod \"payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd\" (UID: \"73b09150-fdf8-4bd2-a006-661b24637975\") " pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:06.203395 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.203374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbdf001-13b2-494f-9220-c8edbdc09c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78\" (UID: \"1dbdf001-13b2-494f-9220-c8edbdc09c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:06.211293 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.211261    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4mln\" (UniqueName: \"kubernetes.io/projected/1dbdf001-13b2-494f-9220-c8edbdc09c2f-kube-api-access-c4mln\") pod \"payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78\" (UID: \"1dbdf001-13b2-494f-9220-c8edbdc09c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:06.303074 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.303043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b09150-fdf8-4bd2-a006-661b24637975-plugins-config-volume\") pod \"payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd\" (UID: \"73b09150-fdf8-4bd2-a006-661b24637975\") " pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:06.303260 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.303168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lzlg\" (UniqueName: \"kubernetes.io/projected/73b09150-fdf8-4bd2-a006-661b24637975-kube-api-access-7lzlg\") pod \"payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd\" (UID: \"73b09150-fdf8-4bd2-a006-661b24637975\") " pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:06.303742 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.303722    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b09150-fdf8-4bd2-a006-661b24637975-plugins-config-volume\") pod \"payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd\" (UID: \"73b09150-fdf8-4bd2-a006-661b24637975\") " pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:06.311130 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.311113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lzlg\" (UniqueName: \"kubernetes.io/projected/73b09150-fdf8-4bd2-a006-661b24637975-kube-api-access-7lzlg\") pod \"payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd\" (UID: \"73b09150-fdf8-4bd2-a006-661b24637975\") " pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:06.405127 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.405047    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:06.417064 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.417038    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:06.577733 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.577710    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"]
Aug 04 21:13:06.579781 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:13:06.579756    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b09150_fdf8_4bd2_a006_661b24637975.slice/crio-e695121694c791f3384aa990db475129c466590eeb960f49e20b1c007250481c WatchSource:0}: Error finding container e695121694c791f3384aa990db475129c466590eeb960f49e20b1c007250481c: Status 404 returned error can't find the container with id e695121694c791f3384aa990db475129c466590eeb960f49e20b1c007250481c
Aug 04 21:13:06.760171 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.760141    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"]
Aug 04 21:13:06.762311 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:13:06.762281    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbdf001_13b2_494f_9220_c8edbdc09c2f.slice/crio-e92942da6da0f963d253c24bbd522c325d3cf1d942365f32be742064ebfba890 WatchSource:0}: Error finding container e92942da6da0f963d253c24bbd522c325d3cf1d942365f32be742064ebfba890: Status 404 returned error can't find the container with id e92942da6da0f963d253c24bbd522c325d3cf1d942365f32be742064ebfba890
Aug 04 21:13:06.893424 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.893393    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd" event={"ID":"73b09150-fdf8-4bd2-a006-661b24637975","Type":"ContainerStarted","Data":"f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8"}
Aug 04 21:13:06.893567 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.893488    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:06.893567 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.893508    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd" event={"ID":"73b09150-fdf8-4bd2-a006-661b24637975","Type":"ContainerStarted","Data":"e695121694c791f3384aa990db475129c466590eeb960f49e20b1c007250481c"}
Aug 04 21:13:06.894859 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.894837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78" event={"ID":"1dbdf001-13b2-494f-9220-c8edbdc09c2f","Type":"ContainerStarted","Data":"70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6"}
Aug 04 21:13:06.894956 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.894883    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:06.894956 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.894898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78" event={"ID":"1dbdf001-13b2-494f-9220-c8edbdc09c2f","Type":"ContainerStarted","Data":"e92942da6da0f963d253c24bbd522c325d3cf1d942365f32be742064ebfba890"}
Aug 04 21:13:06.910950 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.910905    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd" podStartSLOduration=0.910892794 podStartE2EDuration="910.892794ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:13:06.908799541 +0000 UTC m=+1961.660905519" watchObservedRunningTime="2026-08-04 21:13:06.910892794 +0000 UTC m=+1961.662998772"
Aug 04 21:13:06.927501 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:06.927464    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78" podStartSLOduration=0.927451426 podStartE2EDuration="927.451426ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:13:06.925444805 +0000 UTC m=+1961.677550782" watchObservedRunningTime="2026-08-04 21:13:06.927451426 +0000 UTC m=+1961.679557404"
Aug 04 21:13:08.951593 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:08.951560    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:13:17.901539 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:17.901508    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:17.901961 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:17.901819    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:36.153671 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.153638    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"]
Aug 04 21:13:36.154275 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.153910    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd" podUID="73b09150-fdf8-4bd2-a006-661b24637975" containerName="payload-processing" containerID="cri-o://f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8" gracePeriod=30
Aug 04 21:13:36.161976 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.161949    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"]
Aug 04 21:13:36.162207 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.162157    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78" podUID="1dbdf001-13b2-494f-9220-c8edbdc09c2f" containerName="payload-pre-processing" containerID="cri-o://70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6" gracePeriod=30
Aug 04 21:13:36.408991 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.408932    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:36.434928 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.434902    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:36.564563 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.564536    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4mln\" (UniqueName: \"kubernetes.io/projected/1dbdf001-13b2-494f-9220-c8edbdc09c2f-kube-api-access-c4mln\") pod \"1dbdf001-13b2-494f-9220-c8edbdc09c2f\" (UID: \"1dbdf001-13b2-494f-9220-c8edbdc09c2f\") "
Aug 04 21:13:36.564706 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.564620    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lzlg\" (UniqueName: \"kubernetes.io/projected/73b09150-fdf8-4bd2-a006-661b24637975-kube-api-access-7lzlg\") pod \"73b09150-fdf8-4bd2-a006-661b24637975\" (UID: \"73b09150-fdf8-4bd2-a006-661b24637975\") "
Aug 04 21:13:36.564706 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.564682    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b09150-fdf8-4bd2-a006-661b24637975-plugins-config-volume\") pod \"73b09150-fdf8-4bd2-a006-661b24637975\" (UID: \"73b09150-fdf8-4bd2-a006-661b24637975\") "
Aug 04 21:13:36.564706 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.564702    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbdf001-13b2-494f-9220-c8edbdc09c2f-plugins-config-volume\") pod \"1dbdf001-13b2-494f-9220-c8edbdc09c2f\" (UID: \"1dbdf001-13b2-494f-9220-c8edbdc09c2f\") "
Aug 04 21:13:36.565067 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.565042    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b09150-fdf8-4bd2-a006-661b24637975-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73b09150-fdf8-4bd2-a006-661b24637975" (UID: "73b09150-fdf8-4bd2-a006-661b24637975"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:13:36.565151 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.565096    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbdf001-13b2-494f-9220-c8edbdc09c2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dbdf001-13b2-494f-9220-c8edbdc09c2f" (UID: "1dbdf001-13b2-494f-9220-c8edbdc09c2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:13:36.566743 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.566589    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbdf001-13b2-494f-9220-c8edbdc09c2f-kube-api-access-c4mln" (OuterVolumeSpecName: "kube-api-access-c4mln") pod "1dbdf001-13b2-494f-9220-c8edbdc09c2f" (UID: "1dbdf001-13b2-494f-9220-c8edbdc09c2f"). InnerVolumeSpecName "kube-api-access-c4mln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:13:36.566743 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.566657    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b09150-fdf8-4bd2-a006-661b24637975-kube-api-access-7lzlg" (OuterVolumeSpecName: "kube-api-access-7lzlg") pod "73b09150-fdf8-4bd2-a006-661b24637975" (UID: "73b09150-fdf8-4bd2-a006-661b24637975"). InnerVolumeSpecName "kube-api-access-7lzlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:13:36.665699 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.665633    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b09150-fdf8-4bd2-a006-661b24637975-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:13:36.665699 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.665658    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbdf001-13b2-494f-9220-c8edbdc09c2f-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:13:36.665699 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.665668    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4mln\" (UniqueName: \"kubernetes.io/projected/1dbdf001-13b2-494f-9220-c8edbdc09c2f-kube-api-access-c4mln\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:13:36.665699 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:36.665678    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lzlg\" (UniqueName: \"kubernetes.io/projected/73b09150-fdf8-4bd2-a006-661b24637975-kube-api-access-7lzlg\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:13:37.050332 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.050299    2565 generic.go:356] "Generic (PLEG): container finished" podID="73b09150-fdf8-4bd2-a006-661b24637975" containerID="f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8" exitCode=0
Aug 04 21:13:37.050516 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.050375    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"
Aug 04 21:13:37.050516 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.050395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd" event={"ID":"73b09150-fdf8-4bd2-a006-661b24637975","Type":"ContainerDied","Data":"f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8"}
Aug 04 21:13:37.050516 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.050452    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd" event={"ID":"73b09150-fdf8-4bd2-a006-661b24637975","Type":"ContainerDied","Data":"e695121694c791f3384aa990db475129c466590eeb960f49e20b1c007250481c"}
Aug 04 21:13:37.050516 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.050480    2565 scope.go:117] "RemoveContainer" containerID="f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8"
Aug 04 21:13:37.051801 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.051781    2565 generic.go:356] "Generic (PLEG): container finished" podID="1dbdf001-13b2-494f-9220-c8edbdc09c2f" containerID="70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6" exitCode=0
Aug 04 21:13:37.051900 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.051840    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"
Aug 04 21:13:37.051900 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.051842    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78" event={"ID":"1dbdf001-13b2-494f-9220-c8edbdc09c2f","Type":"ContainerDied","Data":"70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6"}
Aug 04 21:13:37.051900 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.051878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78" event={"ID":"1dbdf001-13b2-494f-9220-c8edbdc09c2f","Type":"ContainerDied","Data":"e92942da6da0f963d253c24bbd522c325d3cf1d942365f32be742064ebfba890"}
Aug 04 21:13:37.060766 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.060748    2565 scope.go:117] "RemoveContainer" containerID="f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8"
Aug 04 21:13:37.061024 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:13:37.061002    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8\": container with ID starting with f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8 not found: ID does not exist" containerID="f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8"
Aug 04 21:13:37.061147 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.061030    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8"} err="failed to get container status \"f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8\": rpc error: code = NotFound desc = could not find container \"f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8\": container with ID starting with f02d06ea48118616103ee17370eaba5174bd50c2090d4cbf19dd017d84acd8f8 not found: ID does not exist"
Aug 04 21:13:37.061147 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.061048    2565 scope.go:117] "RemoveContainer" containerID="70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6"
Aug 04 21:13:37.068532 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.068516    2565 scope.go:117] "RemoveContainer" containerID="70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6"
Aug 04 21:13:37.068791 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:13:37.068774    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6\": container with ID starting with 70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6 not found: ID does not exist" containerID="70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6"
Aug 04 21:13:37.068845 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.068795    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6"} err="failed to get container status \"70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6\": rpc error: code = NotFound desc = could not find container \"70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6\": container with ID starting with 70e2ff37d200c54345e4cafc6e962f915f2983de4e52b6ca4d66ba51741587d6 not found: ID does not exist"
Aug 04 21:13:37.073955 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.073932    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"]
Aug 04 21:13:37.078612 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.078593    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-043d8f1f-567f86c798-mz9fd"]
Aug 04 21:13:37.089268 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.089246    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"]
Aug 04 21:13:37.095561 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.095538    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-043d8f1f-74469cd7fb-n9h78"]
Aug 04 21:13:37.760737 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.760705    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dbdf001-13b2-494f-9220-c8edbdc09c2f" path="/var/lib/kubelet/pods/1dbdf001-13b2-494f-9220-c8edbdc09c2f/volumes"
Aug 04 21:13:37.761107 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:37.761065    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73b09150-fdf8-4bd2-a006-661b24637975" path="/var/lib/kubelet/pods/73b09150-fdf8-4bd2-a006-661b24637975/volumes"
Aug 04 21:13:39.050225 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:39.050193    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:13:50.551634 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:13:50.551596    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:15:21.146920 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:21.146883    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:15:25.864086 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:25.864055    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:15:25.866671 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:25.866649    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:15:30.879512 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:30.879485    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:15:41.234413 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.234372    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"]
Aug 04 21:15:41.234921 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.234884    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbdf001-13b2-494f-9220-c8edbdc09c2f" containerName="payload-pre-processing"
Aug 04 21:15:41.234921 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.234902    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbdf001-13b2-494f-9220-c8edbdc09c2f" containerName="payload-pre-processing"
Aug 04 21:15:41.234921 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.234921    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b09150-fdf8-4bd2-a006-661b24637975" containerName="payload-processing"
Aug 04 21:15:41.235026 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.234927    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b09150-fdf8-4bd2-a006-661b24637975" containerName="payload-processing"
Aug 04 21:15:41.235026 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.235003    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73b09150-fdf8-4bd2-a006-661b24637975" containerName="payload-processing"
Aug 04 21:15:41.235026 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.235017    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbdf001-13b2-494f-9220-c8edbdc09c2f" containerName="payload-pre-processing"
Aug 04 21:15:41.238004 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.237986    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:15:41.240662 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.240639    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2fcd204a\""
Aug 04 21:15:41.240785 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.240720    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2fcd204a-dockercfg-b6ldr\""
Aug 04 21:15:41.249193 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.249162    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"]
Aug 04 21:15:41.297219 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.297185    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"]
Aug 04 21:15:41.300596 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.300579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:15:41.310752 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.310726    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"]
Aug 04 21:15:41.357679 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.357651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xq78\" (UniqueName: \"kubernetes.io/projected/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-kube-api-access-7xq78\") pod \"payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99\" (UID: \"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:15:41.358016 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.357685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f235d54e-d85f-4005-b753-d1354ff20b72-plugins-config-volume\") pod \"payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc\" (UID: \"f235d54e-d85f-4005-b753-d1354ff20b72\") " pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:15:41.358016 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.357710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flmrb\" (UniqueName: \"kubernetes.io/projected/f235d54e-d85f-4005-b753-d1354ff20b72-kube-api-access-flmrb\") pod \"payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc\" (UID: \"f235d54e-d85f-4005-b753-d1354ff20b72\") " pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:15:41.358016 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.357837    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99\" (UID: \"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:15:41.459070 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.459040    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xq78\" (UniqueName: \"kubernetes.io/projected/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-kube-api-access-7xq78\") pod \"payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99\" (UID: \"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:15:41.459070 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.459073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f235d54e-d85f-4005-b753-d1354ff20b72-plugins-config-volume\") pod \"payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc\" (UID: \"f235d54e-d85f-4005-b753-d1354ff20b72\") " pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:15:41.459290 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.459090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flmrb\" (UniqueName: \"kubernetes.io/projected/f235d54e-d85f-4005-b753-d1354ff20b72-kube-api-access-flmrb\") pod \"payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc\" (UID: \"f235d54e-d85f-4005-b753-d1354ff20b72\") " pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:15:41.459290 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.459160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99\" (UID: \"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:15:41.460014 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.459989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99\" (UID: \"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:15:41.460069 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.459989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f235d54e-d85f-4005-b753-d1354ff20b72-plugins-config-volume\") pod \"payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc\" (UID: \"f235d54e-d85f-4005-b753-d1354ff20b72\") " pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:15:41.468089 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.468068    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flmrb\" (UniqueName: \"kubernetes.io/projected/f235d54e-d85f-4005-b753-d1354ff20b72-kube-api-access-flmrb\") pod \"payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc\" (UID: \"f235d54e-d85f-4005-b753-d1354ff20b72\") " pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:15:41.468196 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.468094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xq78\" (UniqueName: \"kubernetes.io/projected/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-kube-api-access-7xq78\") pod \"payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99\" (UID: \"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:15:41.548901 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.548819    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:15:41.610578 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.610534    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:15:41.690760 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.690734    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"]
Aug 04 21:15:41.693133 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:15:41.693105    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf235d54e_d85f_4005_b753_d1354ff20b72.slice/crio-5408bf456610f0a3a6fc40faad24cf6d48c8e570ecdae0f535cf21bb50845c2d WatchSource:0}: Error finding container 5408bf456610f0a3a6fc40faad24cf6d48c8e570ecdae0f535cf21bb50845c2d: Status 404 returned error can't find the container with id 5408bf456610f0a3a6fc40faad24cf6d48c8e570ecdae0f535cf21bb50845c2d
Aug 04 21:15:41.757429 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:15:41.757401    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfc4822_ecd2_47bf_ae8e_0e0718a05aeb.slice/crio-22f9bdbdf07e12d86ac257c6f88688ce657f4c3ec94aeb56d8009ae59170d7b8 WatchSource:0}: Error finding container 22f9bdbdf07e12d86ac257c6f88688ce657f4c3ec94aeb56d8009ae59170d7b8: Status 404 returned error can't find the container with id 22f9bdbdf07e12d86ac257c6f88688ce657f4c3ec94aeb56d8009ae59170d7b8
Aug 04 21:15:41.760916 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:41.760891    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"]
Aug 04 21:15:42.642755 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:42.642686    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc" event={"ID":"f235d54e-d85f-4005-b753-d1354ff20b72","Type":"ContainerStarted","Data":"fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956"}
Aug 04 21:15:42.642755 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:42.642755    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:15:42.643272 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:42.642781    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc" event={"ID":"f235d54e-d85f-4005-b753-d1354ff20b72","Type":"ContainerStarted","Data":"5408bf456610f0a3a6fc40faad24cf6d48c8e570ecdae0f535cf21bb50845c2d"}
Aug 04 21:15:42.644372 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:42.644328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99" event={"ID":"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb","Type":"ContainerStarted","Data":"8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771"}
Aug 04 21:15:42.644464 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:42.644399    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:15:42.644464 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:42.644417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99" event={"ID":"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb","Type":"ContainerStarted","Data":"22f9bdbdf07e12d86ac257c6f88688ce657f4c3ec94aeb56d8009ae59170d7b8"}
Aug 04 21:15:42.661497 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:42.661452    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc" podStartSLOduration=1.6614168280000001 podStartE2EDuration="1.661416828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:15:42.658793319 +0000 UTC m=+2117.410899298" watchObservedRunningTime="2026-08-04 21:15:42.661416828 +0000 UTC m=+2117.413522804"
Aug 04 21:15:42.675247 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:42.675201    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99" podStartSLOduration=1.675190553 podStartE2EDuration="1.675190553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:15:42.673609165 +0000 UTC m=+2117.425715146" watchObservedRunningTime="2026-08-04 21:15:42.675190553 +0000 UTC m=+2117.427296586"
Aug 04 21:15:43.747583 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:43.747550    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:15:53.652904 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:53.652875    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:15:53.653504 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:15:53.652942    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:16:22.324546 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.324450    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"]
Aug 04 21:16:22.324936 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.324708    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc" podUID="f235d54e-d85f-4005-b753-d1354ff20b72" containerName="payload-processing" containerID="cri-o://fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956" gracePeriod=30
Aug 04 21:16:22.331978 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.331709    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"]
Aug 04 21:16:22.331978 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.331934    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99" podUID="bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb" containerName="payload-pre-processing" containerID="cri-o://8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771" gracePeriod=30
Aug 04 21:16:22.590406 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.590382    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:16:22.614685 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.614660    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:16:22.637679 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.637650    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f235d54e-d85f-4005-b753-d1354ff20b72-plugins-config-volume\") pod \"f235d54e-d85f-4005-b753-d1354ff20b72\" (UID: \"f235d54e-d85f-4005-b753-d1354ff20b72\") "
Aug 04 21:16:22.637803 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.637786    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flmrb\" (UniqueName: \"kubernetes.io/projected/f235d54e-d85f-4005-b753-d1354ff20b72-kube-api-access-flmrb\") pod \"f235d54e-d85f-4005-b753-d1354ff20b72\" (UID: \"f235d54e-d85f-4005-b753-d1354ff20b72\") "
Aug 04 21:16:22.638009 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.637987    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f235d54e-d85f-4005-b753-d1354ff20b72-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f235d54e-d85f-4005-b753-d1354ff20b72" (UID: "f235d54e-d85f-4005-b753-d1354ff20b72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:16:22.638115 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.638096    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f235d54e-d85f-4005-b753-d1354ff20b72-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:16:22.639738 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.639716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f235d54e-d85f-4005-b753-d1354ff20b72-kube-api-access-flmrb" (OuterVolumeSpecName: "kube-api-access-flmrb") pod "f235d54e-d85f-4005-b753-d1354ff20b72" (UID: "f235d54e-d85f-4005-b753-d1354ff20b72"). InnerVolumeSpecName "kube-api-access-flmrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:16:22.739186 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.739148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xq78\" (UniqueName: \"kubernetes.io/projected/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-kube-api-access-7xq78\") pod \"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb\" (UID: \"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb\") "
Aug 04 21:16:22.739399 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.739251    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-plugins-config-volume\") pod \"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb\" (UID: \"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb\") "
Aug 04 21:16:22.739576 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.739559    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flmrb\" (UniqueName: \"kubernetes.io/projected/f235d54e-d85f-4005-b753-d1354ff20b72-kube-api-access-flmrb\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:16:22.739639 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.739617    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb" (UID: "bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:16:22.741211 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.741190    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-kube-api-access-7xq78" (OuterVolumeSpecName: "kube-api-access-7xq78") pod "bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb" (UID: "bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb"). InnerVolumeSpecName "kube-api-access-7xq78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:16:22.836574 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.836475    2565 generic.go:356] "Generic (PLEG): container finished" podID="bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb" containerID="8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771" exitCode=0
Aug 04 21:16:22.836574 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.836510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99" event={"ID":"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb","Type":"ContainerDied","Data":"8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771"}
Aug 04 21:16:22.836574 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.836531    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"
Aug 04 21:16:22.836860 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.836577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99" event={"ID":"bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb","Type":"ContainerDied","Data":"22f9bdbdf07e12d86ac257c6f88688ce657f4c3ec94aeb56d8009ae59170d7b8"}
Aug 04 21:16:22.836860 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.836608    2565 scope.go:117] "RemoveContainer" containerID="8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771"
Aug 04 21:16:22.838003 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.837965    2565 generic.go:356] "Generic (PLEG): container finished" podID="f235d54e-d85f-4005-b753-d1354ff20b72" containerID="fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956" exitCode=0
Aug 04 21:16:22.838114 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.837990    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc" event={"ID":"f235d54e-d85f-4005-b753-d1354ff20b72","Type":"ContainerDied","Data":"fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956"}
Aug 04 21:16:22.838114 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.838033    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"
Aug 04 21:16:22.838114 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.838047    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc" event={"ID":"f235d54e-d85f-4005-b753-d1354ff20b72","Type":"ContainerDied","Data":"5408bf456610f0a3a6fc40faad24cf6d48c8e570ecdae0f535cf21bb50845c2d"}
Aug 04 21:16:22.840241 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.840184    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xq78\" (UniqueName: \"kubernetes.io/projected/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-kube-api-access-7xq78\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:16:22.840241 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.840208    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:16:22.846659 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.846639    2565 scope.go:117] "RemoveContainer" containerID="8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771"
Aug 04 21:16:22.846966 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:16:22.846936    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771\": container with ID starting with 8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771 not found: ID does not exist" containerID="8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771"
Aug 04 21:16:22.847060 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.846975    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771"} err="failed to get container status \"8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771\": rpc error: code = NotFound desc = could not find container \"8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771\": container with ID starting with 8690b12ff3957578a5d8cbeff74510cfb770ab90e37ef0b0c7f760556a33b771 not found: ID does not exist"
Aug 04 21:16:22.847060 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.846999    2565 scope.go:117] "RemoveContainer" containerID="fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956"
Aug 04 21:16:22.856286 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.856267    2565 scope.go:117] "RemoveContainer" containerID="fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956"
Aug 04 21:16:22.856589 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:16:22.856571    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956\": container with ID starting with fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956 not found: ID does not exist" containerID="fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956"
Aug 04 21:16:22.856646 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.856595    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956"} err="failed to get container status \"fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956\": rpc error: code = NotFound desc = could not find container \"fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956\": container with ID starting with fdc954132e3d241eb51f3cdebcba6ed6bc070ca5d1f25a49589c5aa9a1134956 not found: ID does not exist"
Aug 04 21:16:22.866173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.866152    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"]
Aug 04 21:16:22.870868 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.870848    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fcd204a-7785fc9c48-bck99"]
Aug 04 21:16:22.882388 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.882366    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"]
Aug 04 21:16:22.885490 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:22.885456    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fcd204a-c8d558b8b-8b4kc"]
Aug 04 21:16:23.760502 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:23.760473    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb" path="/var/lib/kubelet/pods/bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb/volumes"
Aug 04 21:16:23.760883 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:23.760845    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f235d54e-d85f-4005-b753-d1354ff20b72" path="/var/lib/kubelet/pods/f235d54e-d85f-4005-b753-d1354ff20b72/volumes"
Aug 04 21:16:24.346801 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:16:24.346765    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:17:01.759722 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:17:01.759693    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:18:47.855410 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:47.855376    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:18:50.240760 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.240730    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"]
Aug 04 21:18:50.241146 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.241107    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f235d54e-d85f-4005-b753-d1354ff20b72" containerName="payload-processing"
Aug 04 21:18:50.241146 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.241119    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f235d54e-d85f-4005-b753-d1354ff20b72" containerName="payload-processing"
Aug 04 21:18:50.241146 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.241136    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb" containerName="payload-pre-processing"
Aug 04 21:18:50.241146 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.241142    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb" containerName="payload-pre-processing"
Aug 04 21:18:50.241297 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.241211    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbfc4822-ecd2-47bf-ae8e-0e0718a05aeb" containerName="payload-pre-processing"
Aug 04 21:18:50.241297 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.241222    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f235d54e-d85f-4005-b753-d1354ff20b72" containerName="payload-processing"
Aug 04 21:18:50.244415 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.244397    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:18:50.246688 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.246665    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d1f5ec52\""
Aug 04 21:18:50.255671 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.255651    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"]
Aug 04 21:18:50.266843 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.266822    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"]
Aug 04 21:18:50.270245 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.270227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:18:50.273128 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.273108    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d1f5ec52-dockercfg-9dmkd\""
Aug 04 21:18:50.285827 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.285801    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"]
Aug 04 21:18:50.340636 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.340608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fc7ec8-b635-4ee8-a94c-3650b21493e3-plugins-config-volume\") pod \"payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt\" (UID: \"40fc7ec8-b635-4ee8-a94c-3650b21493e3\") " pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:18:50.340793 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.340662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfzx\" (UniqueName: \"kubernetes.io/projected/40fc7ec8-b635-4ee8-a94c-3650b21493e3-kube-api-access-bxfzx\") pod \"payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt\" (UID: \"40fc7ec8-b635-4ee8-a94c-3650b21493e3\") " pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:18:50.340793 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.340725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6871538-67c5-4045-82a5-668b65eccb00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4\" (UID: \"b6871538-67c5-4045-82a5-668b65eccb00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:18:50.340793 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.340771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n9nk\" (UniqueName: \"kubernetes.io/projected/b6871538-67c5-4045-82a5-668b65eccb00-kube-api-access-7n9nk\") pod \"payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4\" (UID: \"b6871538-67c5-4045-82a5-668b65eccb00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:18:50.441629 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.441595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fc7ec8-b635-4ee8-a94c-3650b21493e3-plugins-config-volume\") pod \"payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt\" (UID: \"40fc7ec8-b635-4ee8-a94c-3650b21493e3\") " pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:18:50.441778 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.441655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfzx\" (UniqueName: \"kubernetes.io/projected/40fc7ec8-b635-4ee8-a94c-3650b21493e3-kube-api-access-bxfzx\") pod \"payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt\" (UID: \"40fc7ec8-b635-4ee8-a94c-3650b21493e3\") " pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:18:50.441778 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.441698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6871538-67c5-4045-82a5-668b65eccb00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4\" (UID: \"b6871538-67c5-4045-82a5-668b65eccb00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:18:50.441778 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.441731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9nk\" (UniqueName: \"kubernetes.io/projected/b6871538-67c5-4045-82a5-668b65eccb00-kube-api-access-7n9nk\") pod \"payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4\" (UID: \"b6871538-67c5-4045-82a5-668b65eccb00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:18:50.442433 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.442413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fc7ec8-b635-4ee8-a94c-3650b21493e3-plugins-config-volume\") pod \"payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt\" (UID: \"40fc7ec8-b635-4ee8-a94c-3650b21493e3\") " pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:18:50.442484 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.442434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6871538-67c5-4045-82a5-668b65eccb00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4\" (UID: \"b6871538-67c5-4045-82a5-668b65eccb00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:18:50.450571 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.450548    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9nk\" (UniqueName: \"kubernetes.io/projected/b6871538-67c5-4045-82a5-668b65eccb00-kube-api-access-7n9nk\") pod \"payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4\" (UID: \"b6871538-67c5-4045-82a5-668b65eccb00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:18:50.450571 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.450586    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfzx\" (UniqueName: \"kubernetes.io/projected/40fc7ec8-b635-4ee8-a94c-3650b21493e3-kube-api-access-bxfzx\") pod \"payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt\" (UID: \"40fc7ec8-b635-4ee8-a94c-3650b21493e3\") " pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:18:50.556214 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.556113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:18:50.582243 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.582213    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:18:50.699687 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.699662    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"]
Aug 04 21:18:50.701547 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:18:50.701522    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fc7ec8_b635_4ee8_a94c_3650b21493e3.slice/crio-3adaee4190be0354c44a3055223f445259b98303da4300809b0b8c114cac01e7 WatchSource:0}: Error finding container 3adaee4190be0354c44a3055223f445259b98303da4300809b0b8c114cac01e7: Status 404 returned error can't find the container with id 3adaee4190be0354c44a3055223f445259b98303da4300809b0b8c114cac01e7
Aug 04 21:18:50.735794 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:50.735772    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"]
Aug 04 21:18:50.738467 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:18:50.738441    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6871538_67c5_4045_82a5_668b65eccb00.slice/crio-b587349bf2269030ba517408e241966afbad12354075f89cb612f6d451c41945 WatchSource:0}: Error finding container b587349bf2269030ba517408e241966afbad12354075f89cb612f6d451c41945: Status 404 returned error can't find the container with id b587349bf2269030ba517408e241966afbad12354075f89cb612f6d451c41945
Aug 04 21:18:51.549807 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:51.549769    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt" event={"ID":"40fc7ec8-b635-4ee8-a94c-3650b21493e3","Type":"ContainerStarted","Data":"aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc"}
Aug 04 21:18:51.550278 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:51.549841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt" event={"ID":"40fc7ec8-b635-4ee8-a94c-3650b21493e3","Type":"ContainerStarted","Data":"3adaee4190be0354c44a3055223f445259b98303da4300809b0b8c114cac01e7"}
Aug 04 21:18:51.550278 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:51.549938    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:18:51.551641 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:51.551612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4" event={"ID":"b6871538-67c5-4045-82a5-668b65eccb00","Type":"ContainerStarted","Data":"476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7"}
Aug 04 21:18:51.551766 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:51.551657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4" event={"ID":"b6871538-67c5-4045-82a5-668b65eccb00","Type":"ContainerStarted","Data":"b587349bf2269030ba517408e241966afbad12354075f89cb612f6d451c41945"}
Aug 04 21:18:51.551766 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:51.551686    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:18:51.566763 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:51.566722    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt" podStartSLOduration=1.566709492 podStartE2EDuration="1.566709492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:18:51.564863286 +0000 UTC m=+2306.316969264" watchObservedRunningTime="2026-08-04 21:18:51.566709492 +0000 UTC m=+2306.318815469"
Aug 04 21:18:51.582082 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:51.582040    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4" podStartSLOduration=1.582027712 podStartE2EDuration="1.582027712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:18:51.579949321 +0000 UTC m=+2306.332055297" watchObservedRunningTime="2026-08-04 21:18:51.582027712 +0000 UTC m=+2306.334133689"
Aug 04 21:18:54.944093 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:18:54.944060    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:19:02.558455 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:02.558420    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:19:02.559033 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:02.558506    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:19:20.951940 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:20.951860    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"]
Aug 04 21:19:20.955572 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:20.955552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:20.957847 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:20.957824    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d1f5ec52\"/\"openshift-service-ca.crt\""
Aug 04 21:19:20.957924 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:20.957843    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d1f5ec52\"/\"kube-root-ca.crt\""
Aug 04 21:19:20.957924 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:20.957900    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d1f5ec52\"/\"default-dockercfg-cd89r\""
Aug 04 21:19:20.958891 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:20.958873    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d1f5ec52\"/\"e2e-lifecycle-model-d1f5ec52-kserve-self-signed-certs\""
Aug 04 21:19:20.965842 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:20.965820    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"]
Aug 04 21:19:21.011279 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.011253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-dshm\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.011490 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.011297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-tmp-dir\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.011490 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.011316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed3a3df-215d-4e41-a26b-109c85a73de8-tls-certs\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.011490 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.011423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-home\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.011490 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.011478    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2bv6\" (UniqueName: \"kubernetes.io/projected/7ed3a3df-215d-4e41-a26b-109c85a73de8-kube-api-access-f2bv6\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.011683 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.011557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-model-cache\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.112534 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.112494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-home\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.112534 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.112543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bv6\" (UniqueName: \"kubernetes.io/projected/7ed3a3df-215d-4e41-a26b-109c85a73de8-kube-api-access-f2bv6\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.112798 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.112598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-model-cache\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.112798 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.112626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-dshm\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.112798 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.112673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-tmp-dir\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.112798 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.112696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed3a3df-215d-4e41-a26b-109c85a73de8-tls-certs\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.113018 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.112988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-home\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.113067 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.113040    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-model-cache\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.113067 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.113055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-tmp-dir\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.114917 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.114894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-dshm\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.115168 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.115151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed3a3df-215d-4e41-a26b-109c85a73de8-tls-certs\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.120705 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.120682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2bv6\" (UniqueName: \"kubernetes.io/projected/7ed3a3df-215d-4e41-a26b-109c85a73de8-kube-api-access-f2bv6\") pod \"e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") " pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.266764 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.266725    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:21.406255 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.406231    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"]
Aug 04 21:19:21.412460 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:19:21.412425    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ed3a3df_215d_4e41_a26b_109c85a73de8.slice/crio-85de68ab7cad49fdb9d300a5632cb2fb56972e83995a757f04aaa70b0cb0f632 WatchSource:0}: Error finding container 85de68ab7cad49fdb9d300a5632cb2fb56972e83995a757f04aaa70b0cb0f632: Status 404 returned error can't find the container with id 85de68ab7cad49fdb9d300a5632cb2fb56972e83995a757f04aaa70b0cb0f632
Aug 04 21:19:21.703805 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.703720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g" event={"ID":"7ed3a3df-215d-4e41-a26b-109c85a73de8","Type":"ContainerStarted","Data":"3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703"}
Aug 04 21:19:21.703805 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.703771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g" event={"ID":"7ed3a3df-215d-4e41-a26b-109c85a73de8","Type":"ContainerStarted","Data":"85de68ab7cad49fdb9d300a5632cb2fb56972e83995a757f04aaa70b0cb0f632"}
Aug 04 21:19:21.722322 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:21.722273    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g" podStartSLOduration=1.72225877 podStartE2EDuration="1.72225877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:19:21.720117791 +0000 UTC m=+2336.472223778" watchObservedRunningTime="2026-08-04 21:19:21.72225877 +0000 UTC m=+2336.474364800"
Aug 04 21:19:23.151313 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:23.151266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:19:31.267197 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:31.267157    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:31.267627 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:31.267216    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:31.280174 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:31.280146    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:31.762223 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:31.762193    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:19:37.554709 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:19:37.554675    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:20:00.743095 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:00.743065    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:20:03.305635 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.305595    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"]
Aug 04 21:20:03.307503 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.307465    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt" podUID="40fc7ec8-b635-4ee8-a94c-3650b21493e3" containerName="payload-processing" containerID="cri-o://aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc" gracePeriod=30
Aug 04 21:20:03.309266 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.308722    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"]
Aug 04 21:20:03.309266 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.309000    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4" podUID="b6871538-67c5-4045-82a5-668b65eccb00" containerName="payload-pre-processing" containerID="cri-o://476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7" gracePeriod=30
Aug 04 21:20:03.583204 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.583179    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:20:03.586483 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.586463    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:20:03.695108 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.695072    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n9nk\" (UniqueName: \"kubernetes.io/projected/b6871538-67c5-4045-82a5-668b65eccb00-kube-api-access-7n9nk\") pod \"b6871538-67c5-4045-82a5-668b65eccb00\" (UID: \"b6871538-67c5-4045-82a5-668b65eccb00\") "
Aug 04 21:20:03.695316 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.695132    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxfzx\" (UniqueName: \"kubernetes.io/projected/40fc7ec8-b635-4ee8-a94c-3650b21493e3-kube-api-access-bxfzx\") pod \"40fc7ec8-b635-4ee8-a94c-3650b21493e3\" (UID: \"40fc7ec8-b635-4ee8-a94c-3650b21493e3\") "
Aug 04 21:20:03.695316 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.695170    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fc7ec8-b635-4ee8-a94c-3650b21493e3-plugins-config-volume\") pod \"40fc7ec8-b635-4ee8-a94c-3650b21493e3\" (UID: \"40fc7ec8-b635-4ee8-a94c-3650b21493e3\") "
Aug 04 21:20:03.695496 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.695328    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6871538-67c5-4045-82a5-668b65eccb00-plugins-config-volume\") pod \"b6871538-67c5-4045-82a5-668b65eccb00\" (UID: \"b6871538-67c5-4045-82a5-668b65eccb00\") "
Aug 04 21:20:03.695557 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.695508    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fc7ec8-b635-4ee8-a94c-3650b21493e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40fc7ec8-b635-4ee8-a94c-3650b21493e3" (UID: "40fc7ec8-b635-4ee8-a94c-3650b21493e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:20:03.695708 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.695686    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6871538-67c5-4045-82a5-668b65eccb00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6871538-67c5-4045-82a5-668b65eccb00" (UID: "b6871538-67c5-4045-82a5-668b65eccb00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:20:03.695812 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.695796    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fc7ec8-b635-4ee8-a94c-3650b21493e3-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:20:03.695858 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.695817    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6871538-67c5-4045-82a5-668b65eccb00-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:20:03.697205 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.697182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6871538-67c5-4045-82a5-668b65eccb00-kube-api-access-7n9nk" (OuterVolumeSpecName: "kube-api-access-7n9nk") pod "b6871538-67c5-4045-82a5-668b65eccb00" (UID: "b6871538-67c5-4045-82a5-668b65eccb00"). InnerVolumeSpecName "kube-api-access-7n9nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:20:03.697291 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.697255    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fc7ec8-b635-4ee8-a94c-3650b21493e3-kube-api-access-bxfzx" (OuterVolumeSpecName: "kube-api-access-bxfzx") pod "40fc7ec8-b635-4ee8-a94c-3650b21493e3" (UID: "40fc7ec8-b635-4ee8-a94c-3650b21493e3"). InnerVolumeSpecName "kube-api-access-bxfzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:20:03.796878 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.796845    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n9nk\" (UniqueName: \"kubernetes.io/projected/b6871538-67c5-4045-82a5-668b65eccb00-kube-api-access-7n9nk\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:20:03.796878 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.796873    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxfzx\" (UniqueName: \"kubernetes.io/projected/40fc7ec8-b635-4ee8-a94c-3650b21493e3-kube-api-access-bxfzx\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:20:03.901992 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.901906    2565 generic.go:356] "Generic (PLEG): container finished" podID="b6871538-67c5-4045-82a5-668b65eccb00" containerID="476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7" exitCode=0
Aug 04 21:20:03.901992 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.901952    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"
Aug 04 21:20:03.901992 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.901964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4" event={"ID":"b6871538-67c5-4045-82a5-668b65eccb00","Type":"ContainerDied","Data":"476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7"}
Aug 04 21:20:03.902257 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.902019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4" event={"ID":"b6871538-67c5-4045-82a5-668b65eccb00","Type":"ContainerDied","Data":"b587349bf2269030ba517408e241966afbad12354075f89cb612f6d451c41945"}
Aug 04 21:20:03.902257 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.902034    2565 scope.go:117] "RemoveContainer" containerID="476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7"
Aug 04 21:20:03.903370 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.903333    2565 generic.go:356] "Generic (PLEG): container finished" podID="40fc7ec8-b635-4ee8-a94c-3650b21493e3" containerID="aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc" exitCode=0
Aug 04 21:20:03.903468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.903381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt" event={"ID":"40fc7ec8-b635-4ee8-a94c-3650b21493e3","Type":"ContainerDied","Data":"aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc"}
Aug 04 21:20:03.903468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.903397    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"
Aug 04 21:20:03.903468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.903433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt" event={"ID":"40fc7ec8-b635-4ee8-a94c-3650b21493e3","Type":"ContainerDied","Data":"3adaee4190be0354c44a3055223f445259b98303da4300809b0b8c114cac01e7"}
Aug 04 21:20:03.910874 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.910858    2565 scope.go:117] "RemoveContainer" containerID="476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7"
Aug 04 21:20:03.911138 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:20:03.911114    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7\": container with ID starting with 476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7 not found: ID does not exist" containerID="476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7"
Aug 04 21:20:03.911191 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.911145    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7"} err="failed to get container status \"476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7\": rpc error: code = NotFound desc = could not find container \"476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7\": container with ID starting with 476405d42ee2bee22c954c9cff9b85184c1db5ee5537152f929f001fc237e8a7 not found: ID does not exist"
Aug 04 21:20:03.911191 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.911165    2565 scope.go:117] "RemoveContainer" containerID="aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc"
Aug 04 21:20:03.919219 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.919202    2565 scope.go:117] "RemoveContainer" containerID="aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc"
Aug 04 21:20:03.919489 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:20:03.919467    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc\": container with ID starting with aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc not found: ID does not exist" containerID="aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc"
Aug 04 21:20:03.919545 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.919498    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc"} err="failed to get container status \"aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc\": rpc error: code = NotFound desc = could not find container \"aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc\": container with ID starting with aa5a6c6cd674f07d09acc86b2727b8fdbd460547a003d94483b913a58ea06efc not found: ID does not exist"
Aug 04 21:20:03.919857 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.919843    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"]
Aug 04 21:20:03.923377 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.923357    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1f5ec52-6975998cdb-k8xgt"]
Aug 04 21:20:03.934162 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.934141    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"]
Aug 04 21:20:03.937723 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:03.937701    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1f5ec52-55db7d8b68-nphg4"]
Aug 04 21:20:05.760380 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:05.760330    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40fc7ec8-b635-4ee8-a94c-3650b21493e3" path="/var/lib/kubelet/pods/40fc7ec8-b635-4ee8-a94c-3650b21493e3/volumes"
Aug 04 21:20:05.760760 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:05.760726    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6871538-67c5-4045-82a5-668b65eccb00" path="/var/lib/kubelet/pods/b6871538-67c5-4045-82a5-668b65eccb00/volumes"
Aug 04 21:20:06.446645 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:06.446615    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:20:11.659760 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:11.659725    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"]
Aug 04 21:20:25.894632 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:25.894594    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:20:25.896634 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:25.896614    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:20:26.678751 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:26.678693    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g" podUID="7ed3a3df-215d-4e41-a26b-109c85a73de8" containerName="main" containerID="cri-o://3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703" gracePeriod=45
Aug 04 21:20:26.927147 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:26.927121    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:20:27.008492 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.008461    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed3a3df-215d-4e41-a26b-109c85a73de8-tls-certs\") pod \"7ed3a3df-215d-4e41-a26b-109c85a73de8\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") "
Aug 04 21:20:27.008645 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.008498    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2bv6\" (UniqueName: \"kubernetes.io/projected/7ed3a3df-215d-4e41-a26b-109c85a73de8-kube-api-access-f2bv6\") pod \"7ed3a3df-215d-4e41-a26b-109c85a73de8\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") "
Aug 04 21:20:27.008645 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.008543    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-model-cache\") pod \"7ed3a3df-215d-4e41-a26b-109c85a73de8\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") "
Aug 04 21:20:27.008645 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.008589    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-tmp-dir\") pod \"7ed3a3df-215d-4e41-a26b-109c85a73de8\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") "
Aug 04 21:20:27.008645 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.008634    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-dshm\") pod \"7ed3a3df-215d-4e41-a26b-109c85a73de8\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") "
Aug 04 21:20:27.008786 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.008667    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-home\") pod \"7ed3a3df-215d-4e41-a26b-109c85a73de8\" (UID: \"7ed3a3df-215d-4e41-a26b-109c85a73de8\") "
Aug 04 21:20:27.008894 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.008865    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ed3a3df-215d-4e41-a26b-109c85a73de8" (UID: "7ed3a3df-215d-4e41-a26b-109c85a73de8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:20:27.008965 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.008872    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ed3a3df-215d-4e41-a26b-109c85a73de8" (UID: "7ed3a3df-215d-4e41-a26b-109c85a73de8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:20:27.009023 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.008988    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:20:27.009023 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.009016    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:20:27.009105 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.009024    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-home" (OuterVolumeSpecName: "home") pod "7ed3a3df-215d-4e41-a26b-109c85a73de8" (UID: "7ed3a3df-215d-4e41-a26b-109c85a73de8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:20:27.010619 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.010593    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed3a3df-215d-4e41-a26b-109c85a73de8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ed3a3df-215d-4e41-a26b-109c85a73de8" (UID: "7ed3a3df-215d-4e41-a26b-109c85a73de8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 21:20:27.010717 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.010633    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-dshm" (OuterVolumeSpecName: "dshm") pod "7ed3a3df-215d-4e41-a26b-109c85a73de8" (UID: "7ed3a3df-215d-4e41-a26b-109c85a73de8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:20:27.010806 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.010787    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed3a3df-215d-4e41-a26b-109c85a73de8-kube-api-access-f2bv6" (OuterVolumeSpecName: "kube-api-access-f2bv6") pod "7ed3a3df-215d-4e41-a26b-109c85a73de8" (UID: "7ed3a3df-215d-4e41-a26b-109c85a73de8"). InnerVolumeSpecName "kube-api-access-f2bv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:20:27.023959 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.023932    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ed3a3df-215d-4e41-a26b-109c85a73de8" containerID="3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703" exitCode=0
Aug 04 21:20:27.024055 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.023973    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"
Aug 04 21:20:27.024055 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.024004    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g" event={"ID":"7ed3a3df-215d-4e41-a26b-109c85a73de8","Type":"ContainerDied","Data":"3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703"}
Aug 04 21:20:27.024055 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.024037    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g" event={"ID":"7ed3a3df-215d-4e41-a26b-109c85a73de8","Type":"ContainerDied","Data":"85de68ab7cad49fdb9d300a5632cb2fb56972e83995a757f04aaa70b0cb0f632"}
Aug 04 21:20:27.024055 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.024054    2565 scope.go:117] "RemoveContainer" containerID="3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703"
Aug 04 21:20:27.032440 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.032406    2565 scope.go:117] "RemoveContainer" containerID="3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703"
Aug 04 21:20:27.032667 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:20:27.032648    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703\": container with ID starting with 3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703 not found: ID does not exist" containerID="3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703"
Aug 04 21:20:27.032718 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.032675    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703"} err="failed to get container status \"3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703\": rpc error: code = NotFound desc = could not find container \"3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703\": container with ID starting with 3c976b47f2eff5a67d66c84d6470da20371823e504037ad410066819019a6703 not found: ID does not exist"
Aug 04 21:20:27.045721 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.045701    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"]
Aug 04 21:20:27.047299 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.047282    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d1f5ec52/e2e-lifecycle-model-d1f5ec52-kserve-57c7dd8587-f5g7g"]
Aug 04 21:20:27.110249 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.110222    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed3a3df-215d-4e41-a26b-109c85a73de8-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:20:27.110249 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.110246    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2bv6\" (UniqueName: \"kubernetes.io/projected/7ed3a3df-215d-4e41-a26b-109c85a73de8-kube-api-access-f2bv6\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:20:27.110413 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.110257    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:20:27.110413 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.110266    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ed3a3df-215d-4e41-a26b-109c85a73de8-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:20:27.760951 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:27.760912    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ed3a3df-215d-4e41-a26b-109c85a73de8" path="/var/lib/kubelet/pods/7ed3a3df-215d-4e41-a26b-109c85a73de8/volumes"
Aug 04 21:20:57.064614 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:20:57.064532    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:21:08.351649 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:21:08.351612    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:21:15.045486 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:21:15.045451    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:21:23.752825 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:21:23.752790    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:22:05.158468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.155422    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"]
Aug 04 21:22:05.158468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.156261    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6871538-67c5-4045-82a5-668b65eccb00" containerName="payload-pre-processing"
Aug 04 21:22:05.158468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.156289    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6871538-67c5-4045-82a5-668b65eccb00" containerName="payload-pre-processing"
Aug 04 21:22:05.158468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.156331    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40fc7ec8-b635-4ee8-a94c-3650b21493e3" containerName="payload-processing"
Aug 04 21:22:05.158468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.156355    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="40fc7ec8-b635-4ee8-a94c-3650b21493e3" containerName="payload-processing"
Aug 04 21:22:05.158468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.156393    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed3a3df-215d-4e41-a26b-109c85a73de8" containerName="main"
Aug 04 21:22:05.158468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.156402    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed3a3df-215d-4e41-a26b-109c85a73de8" containerName="main"
Aug 04 21:22:05.158468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.156585    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="40fc7ec8-b635-4ee8-a94c-3650b21493e3" containerName="payload-processing"
Aug 04 21:22:05.158468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.156598    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed3a3df-215d-4e41-a26b-109c85a73de8" containerName="main"
Aug 04 21:22:05.158468 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.156617    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6871538-67c5-4045-82a5-668b65eccb00" containerName="payload-pre-processing"
Aug 04 21:22:05.160971 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.160950    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:22:05.163730 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.163522    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-19ad13-dockercfg-568hl\""
Aug 04 21:22:05.163730 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.163537    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-19ad13\""
Aug 04 21:22:05.166764 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.166739    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"]
Aug 04 21:22:05.181394 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.181369    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"]
Aug 04 21:22:05.184741 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.184724    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:22:05.198054 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.198030    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"]
Aug 04 21:22:05.209697 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.209669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230dc476-3051-4d22-9f5d-2c360f1a9df0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4\" (UID: \"230dc476-3051-4d22-9f5d-2c360f1a9df0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:22:05.209807 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.209794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vz29\" (UniqueName: \"kubernetes.io/projected/230dc476-3051-4d22-9f5d-2c360f1a9df0-kube-api-access-7vz29\") pod \"payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4\" (UID: \"230dc476-3051-4d22-9f5d-2c360f1a9df0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:22:05.209854 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.209822    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390a5f44-2fd0-46c5-8a5e-caccce453ed0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8\" (UID: \"390a5f44-2fd0-46c5-8a5e-caccce453ed0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:22:05.209923 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.209902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b8zn\" (UniqueName: \"kubernetes.io/projected/390a5f44-2fd0-46c5-8a5e-caccce453ed0-kube-api-access-7b8zn\") pod \"payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8\" (UID: \"390a5f44-2fd0-46c5-8a5e-caccce453ed0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:22:05.311049 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.311014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vz29\" (UniqueName: \"kubernetes.io/projected/230dc476-3051-4d22-9f5d-2c360f1a9df0-kube-api-access-7vz29\") pod \"payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4\" (UID: \"230dc476-3051-4d22-9f5d-2c360f1a9df0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:22:05.311244 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.311053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390a5f44-2fd0-46c5-8a5e-caccce453ed0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8\" (UID: \"390a5f44-2fd0-46c5-8a5e-caccce453ed0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:22:05.311244 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.311106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8zn\" (UniqueName: \"kubernetes.io/projected/390a5f44-2fd0-46c5-8a5e-caccce453ed0-kube-api-access-7b8zn\") pod \"payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8\" (UID: \"390a5f44-2fd0-46c5-8a5e-caccce453ed0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:22:05.311244 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.311152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230dc476-3051-4d22-9f5d-2c360f1a9df0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4\" (UID: \"230dc476-3051-4d22-9f5d-2c360f1a9df0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:22:05.311918 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.311894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390a5f44-2fd0-46c5-8a5e-caccce453ed0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8\" (UID: \"390a5f44-2fd0-46c5-8a5e-caccce453ed0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:22:05.312011 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.311895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230dc476-3051-4d22-9f5d-2c360f1a9df0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4\" (UID: \"230dc476-3051-4d22-9f5d-2c360f1a9df0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:22:05.319570 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.319543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vz29\" (UniqueName: \"kubernetes.io/projected/230dc476-3051-4d22-9f5d-2c360f1a9df0-kube-api-access-7vz29\") pod \"payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4\" (UID: \"230dc476-3051-4d22-9f5d-2c360f1a9df0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:22:05.319670 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.319640    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8zn\" (UniqueName: \"kubernetes.io/projected/390a5f44-2fd0-46c5-8a5e-caccce453ed0-kube-api-access-7b8zn\") pod \"payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8\" (UID: \"390a5f44-2fd0-46c5-8a5e-caccce453ed0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:22:05.473272 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.473239    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:22:05.494989 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.494964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:22:05.825745 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.825723    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"]
Aug 04 21:22:05.827724 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:22:05.827703    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390a5f44_2fd0_46c5_8a5e_caccce453ed0.slice/crio-7611f0f8f729349267071911f6c9568ad35dbb8bc91d5d81107d4ab2a990283f WatchSource:0}: Error finding container 7611f0f8f729349267071911f6c9568ad35dbb8bc91d5d81107d4ab2a990283f: Status 404 returned error can't find the container with id 7611f0f8f729349267071911f6c9568ad35dbb8bc91d5d81107d4ab2a990283f
Aug 04 21:22:05.860410 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:05.860390    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"]
Aug 04 21:22:05.863490 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:22:05.863466    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230dc476_3051_4d22_9f5d_2c360f1a9df0.slice/crio-9d6d9b4ec891860917ce24c0d4718c253d48760858718584e38380ae1ccaaee8 WatchSource:0}: Error finding container 9d6d9b4ec891860917ce24c0d4718c253d48760858718584e38380ae1ccaaee8: Status 404 returned error can't find the container with id 9d6d9b4ec891860917ce24c0d4718c253d48760858718584e38380ae1ccaaee8
Aug 04 21:22:06.495800 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:06.495760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4" event={"ID":"230dc476-3051-4d22-9f5d-2c360f1a9df0","Type":"ContainerStarted","Data":"122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d"}
Aug 04 21:22:06.496285 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:06.495826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4" event={"ID":"230dc476-3051-4d22-9f5d-2c360f1a9df0","Type":"ContainerStarted","Data":"9d6d9b4ec891860917ce24c0d4718c253d48760858718584e38380ae1ccaaee8"}
Aug 04 21:22:06.496285 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:06.495858    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:22:06.497459 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:06.497428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8" event={"ID":"390a5f44-2fd0-46c5-8a5e-caccce453ed0","Type":"ContainerStarted","Data":"0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4"}
Aug 04 21:22:06.497594 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:06.497483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8" event={"ID":"390a5f44-2fd0-46c5-8a5e-caccce453ed0","Type":"ContainerStarted","Data":"7611f0f8f729349267071911f6c9568ad35dbb8bc91d5d81107d4ab2a990283f"}
Aug 04 21:22:06.497594 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:06.497510    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:22:06.515083 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:06.515036    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4" podStartSLOduration=1.515022474 podStartE2EDuration="1.515022474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:22:06.513472916 +0000 UTC m=+2501.265578894" watchObservedRunningTime="2026-08-04 21:22:06.515022474 +0000 UTC m=+2501.267128450"
Aug 04 21:22:06.530289 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:06.530240    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8" podStartSLOduration=1.530226168 podStartE2EDuration="1.530226168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:22:06.528938445 +0000 UTC m=+2501.281044422" watchObservedRunningTime="2026-08-04 21:22:06.530226168 +0000 UTC m=+2501.282332142"
Aug 04 21:22:07.454350 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:07.454303    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:22:17.504253 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:17.504222    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:22:17.504755 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:17.504411    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:22:33.168289 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.168254    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"]
Aug 04 21:22:33.171756 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.171741    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:22:33.174282 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.174258    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4c3747\""
Aug 04 21:22:33.174433 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.174330    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4c3747-dockercfg-drgpp\""
Aug 04 21:22:33.182166 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.182144    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"]
Aug 04 21:22:33.211302 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.211274    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"]
Aug 04 21:22:33.214581 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.214565    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:22:33.225378 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.225355    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"]
Aug 04 21:22:33.272114 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.272089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7edf508e-bdc1-4833-9c8f-557160016ead-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk\" (UID: \"7edf508e-bdc1-4833-9c8f-557160016ead\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:22:33.272246 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.272133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2clz\" (UniqueName: \"kubernetes.io/projected/7edf508e-bdc1-4833-9c8f-557160016ead-kube-api-access-b2clz\") pod \"payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk\" (UID: \"7edf508e-bdc1-4833-9c8f-557160016ead\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:22:33.373139 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.373098    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b49185-1179-40b1-a819-80dc8ebb551d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg\" (UID: \"24b49185-1179-40b1-a819-80dc8ebb551d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:22:33.373277 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.373153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7edf508e-bdc1-4833-9c8f-557160016ead-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk\" (UID: \"7edf508e-bdc1-4833-9c8f-557160016ead\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:22:33.373277 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.373229    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2clz\" (UniqueName: \"kubernetes.io/projected/7edf508e-bdc1-4833-9c8f-557160016ead-kube-api-access-b2clz\") pod \"payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk\" (UID: \"7edf508e-bdc1-4833-9c8f-557160016ead\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:22:33.373370 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.373275    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r55zk\" (UniqueName: \"kubernetes.io/projected/24b49185-1179-40b1-a819-80dc8ebb551d-kube-api-access-r55zk\") pod \"payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg\" (UID: \"24b49185-1179-40b1-a819-80dc8ebb551d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:22:33.373874 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.373852    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7edf508e-bdc1-4833-9c8f-557160016ead-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk\" (UID: \"7edf508e-bdc1-4833-9c8f-557160016ead\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:22:33.381137 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.381118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2clz\" (UniqueName: \"kubernetes.io/projected/7edf508e-bdc1-4833-9c8f-557160016ead-kube-api-access-b2clz\") pod \"payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk\" (UID: \"7edf508e-bdc1-4833-9c8f-557160016ead\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:22:33.473978 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.473952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r55zk\" (UniqueName: \"kubernetes.io/projected/24b49185-1179-40b1-a819-80dc8ebb551d-kube-api-access-r55zk\") pod \"payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg\" (UID: \"24b49185-1179-40b1-a819-80dc8ebb551d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:22:33.474115 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.474031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b49185-1179-40b1-a819-80dc8ebb551d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg\" (UID: \"24b49185-1179-40b1-a819-80dc8ebb551d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:22:33.474678 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.474662    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b49185-1179-40b1-a819-80dc8ebb551d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg\" (UID: \"24b49185-1179-40b1-a819-80dc8ebb551d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:22:33.481919 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.481869    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:22:33.482190 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.482173    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r55zk\" (UniqueName: \"kubernetes.io/projected/24b49185-1179-40b1-a819-80dc8ebb551d-kube-api-access-r55zk\") pod \"payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg\" (UID: \"24b49185-1179-40b1-a819-80dc8ebb551d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:22:33.525217 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.525186    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:22:33.628833 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.628747    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"]
Aug 04 21:22:33.630365 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:22:33.629336    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edf508e_bdc1_4833_9c8f_557160016ead.slice/crio-cd83bfd0e4caf2f9b0681cfc2b1fbbd7d9db6c42ffc02526d4d845f016386563 WatchSource:0}: Error finding container cd83bfd0e4caf2f9b0681cfc2b1fbbd7d9db6c42ffc02526d4d845f016386563: Status 404 returned error can't find the container with id cd83bfd0e4caf2f9b0681cfc2b1fbbd7d9db6c42ffc02526d4d845f016386563
Aug 04 21:22:33.882487 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:33.882464    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"]
Aug 04 21:22:33.884226 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:22:33.884201    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b49185_1179_40b1_a819_80dc8ebb551d.slice/crio-88cb4a6628cc0ba5af62b0bfdfa7104118900454fb4a590214cdab7fda8a0491 WatchSource:0}: Error finding container 88cb4a6628cc0ba5af62b0bfdfa7104118900454fb4a590214cdab7fda8a0491: Status 404 returned error can't find the container with id 88cb4a6628cc0ba5af62b0bfdfa7104118900454fb4a590214cdab7fda8a0491
Aug 04 21:22:34.641242 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:34.641209    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg" event={"ID":"24b49185-1179-40b1-a819-80dc8ebb551d","Type":"ContainerStarted","Data":"6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e"}
Aug 04 21:22:34.641737 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:34.641639    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:22:34.641737 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:34.641675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg" event={"ID":"24b49185-1179-40b1-a819-80dc8ebb551d","Type":"ContainerStarted","Data":"88cb4a6628cc0ba5af62b0bfdfa7104118900454fb4a590214cdab7fda8a0491"}
Aug 04 21:22:34.642888 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:34.642862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk" event={"ID":"7edf508e-bdc1-4833-9c8f-557160016ead","Type":"ContainerStarted","Data":"8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027"}
Aug 04 21:22:34.643000 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:34.642911    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk" event={"ID":"7edf508e-bdc1-4833-9c8f-557160016ead","Type":"ContainerStarted","Data":"cd83bfd0e4caf2f9b0681cfc2b1fbbd7d9db6c42ffc02526d4d845f016386563"}
Aug 04 21:22:34.643000 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:34.642953    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:22:34.660082 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:34.660040    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg" podStartSLOduration=1.660025069 podStartE2EDuration="1.660025069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:22:34.657800276 +0000 UTC m=+2529.409906255" watchObservedRunningTime="2026-08-04 21:22:34.660025069 +0000 UTC m=+2529.412131046"
Aug 04 21:22:34.674012 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:34.673974    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk" podStartSLOduration=1.6739651370000002 podStartE2EDuration="1.673965137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:22:34.672329842 +0000 UTC m=+2529.424435845" watchObservedRunningTime="2026-08-04 21:22:34.673965137 +0000 UTC m=+2529.426071114"
Aug 04 21:22:35.249746 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:35.249716    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:22:45.653020 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:45.652990    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:22:46.657407 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:22:46.657372    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:23:13.254758 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.254727    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"]
Aug 04 21:23:13.258677 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.258656    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.262583 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.262561    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-19ad13\"/\"test-model-19ad13-kserve-self-signed-certs\""
Aug 04 21:23:13.262740 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.262590    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-19ad13\"/\"default-dockercfg-swnkh\""
Aug 04 21:23:13.262740 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.262597    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-19ad13\"/\"kube-root-ca.crt\""
Aug 04 21:23:13.262740 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.262640    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-19ad13\"/\"openshift-service-ca.crt\""
Aug 04 21:23:13.268949 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.268929    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"]
Aug 04 21:23:13.419878 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.419849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-dshm\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.420043 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.419896    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tmp-dir\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.420043 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.419930    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vb68\" (UniqueName: \"kubernetes.io/projected/2a266ea3-cea0-43ba-af1b-b2bdc435081d-kube-api-access-5vb68\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.420043 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.419981    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tls-certs\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.420167 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.420081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-home\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.420205 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.420175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-model-cache\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.521029 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.520958    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tls-certs\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.521029 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.520994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-home\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.521285 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.521031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-model-cache\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.521285 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.521056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-dshm\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.521285 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.521082    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tmp-dir\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.521285 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.521106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vb68\" (UniqueName: \"kubernetes.io/projected/2a266ea3-cea0-43ba-af1b-b2bdc435081d-kube-api-access-5vb68\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.521542 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.521374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-home\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.521542 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.521436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tmp-dir\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.521542 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.521532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-model-cache\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.523328 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.523308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-dshm\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.523516 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.523497    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tls-certs\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.529828 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.529805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vb68\" (UniqueName: \"kubernetes.io/projected/2a266ea3-cea0-43ba-af1b-b2bdc435081d-kube-api-access-5vb68\") pod \"test-model-19ad13-kserve-757f89987-dsbjn\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") " pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.568650 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.568624    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:13.701752 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.701729    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"]
Aug 04 21:23:13.703323 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:23:13.703296    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a266ea3_cea0_43ba_af1b_b2bdc435081d.slice/crio-fe7c5fccef1ca229f889fb2c067dd7da58b2ce75d6efaa5671087d428781dd58 WatchSource:0}: Error finding container fe7c5fccef1ca229f889fb2c067dd7da58b2ce75d6efaa5671087d428781dd58: Status 404 returned error can't find the container with id fe7c5fccef1ca229f889fb2c067dd7da58b2ce75d6efaa5671087d428781dd58
Aug 04 21:23:13.833754 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.833723    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn" event={"ID":"2a266ea3-cea0-43ba-af1b-b2bdc435081d","Type":"ContainerStarted","Data":"4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74"}
Aug 04 21:23:13.833901 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.833762    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn" event={"ID":"2a266ea3-cea0-43ba-af1b-b2bdc435081d","Type":"ContainerStarted","Data":"fe7c5fccef1ca229f889fb2c067dd7da58b2ce75d6efaa5671087d428781dd58"}
Aug 04 21:23:13.852843 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:13.852796    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn" podStartSLOduration=0.852780442 podStartE2EDuration="852.780442ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:23:13.85144345 +0000 UTC m=+2568.603549428" watchObservedRunningTime="2026-08-04 21:23:13.852780442 +0000 UTC m=+2568.604886415"
Aug 04 21:23:15.450681 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:15.450645    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:23:23.569179 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:23.569145    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:23.569679 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:23.569199    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:23.581648 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:23.581626    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:23.894271 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:23.894198    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:23:24.740856 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:24.740821    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:23:33.447139 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.447103    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"]
Aug 04 21:23:33.453087 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.453066    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.455904 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.455878    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4c3747\"/\"kube-root-ca.crt\""
Aug 04 21:23:33.456896 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.456867    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4c3747\"/\"default-dockercfg-bzp8x\""
Aug 04 21:23:33.456988 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.456967    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4c3747\"/\"test-model-4c3747-kserve-self-signed-certs\""
Aug 04 21:23:33.457140 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.457117    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4c3747\"/\"openshift-service-ca.crt\""
Aug 04 21:23:33.463374 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.463336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"]
Aug 04 21:23:33.501253 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.501224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-home\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.501423 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.501257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6lsm\" (UniqueName: \"kubernetes.io/projected/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-kube-api-access-s6lsm\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.501423 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.501296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-dshm\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.501559 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.501418    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tls-certs\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.501559 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.501516    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tmp-dir\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.501651 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.501565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-model-cache\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.602772 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.602742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tls-certs\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.602938 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.602809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tmp-dir\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.602938 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.602850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-model-cache\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.602938 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.602900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-home\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.602938 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.602932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6lsm\" (UniqueName: \"kubernetes.io/projected/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-kube-api-access-s6lsm\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.603166 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.602975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-dshm\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.603236 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.603171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tmp-dir\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.603299 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.603252    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-model-cache\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.603299 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.603282    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-home\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.605104 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.605083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-dshm\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.605401 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.605384    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tls-certs\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.614249 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.614226    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6lsm\" (UniqueName: \"kubernetes.io/projected/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-kube-api-access-s6lsm\") pod \"test-model-4c3747-kserve-5b77cbb79c-h8bmv\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") " pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.764071 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.764051    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:33.895180 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.895155    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"]
Aug 04 21:23:33.896505 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:23:33.896474    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e745330_e6d1_4be8_b4b6_999bb6a2d60f.slice/crio-50d1c4aa049280c2e2bc182054e00233408503999d67f475fa5fde1b803465a0 WatchSource:0}: Error finding container 50d1c4aa049280c2e2bc182054e00233408503999d67f475fa5fde1b803465a0: Status 404 returned error can't find the container with id 50d1c4aa049280c2e2bc182054e00233408503999d67f475fa5fde1b803465a0
Aug 04 21:23:33.939423 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:33.939392    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv" event={"ID":"6e745330-e6d1-4be8-b4b6-999bb6a2d60f","Type":"ContainerStarted","Data":"50d1c4aa049280c2e2bc182054e00233408503999d67f475fa5fde1b803465a0"}
Aug 04 21:23:34.946404 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:34.946368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv" event={"ID":"6e745330-e6d1-4be8-b4b6-999bb6a2d60f","Type":"ContainerStarted","Data":"8f8655671cdabc50d4320c0674191bd316227cd6164bd5becdaa9ded9c2a5aff"}
Aug 04 21:23:34.965222 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:34.965172    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv" podStartSLOduration=1.9651567490000001 podStartE2EDuration="1.965156749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:23:34.962363684 +0000 UTC m=+2589.714469652" watchObservedRunningTime="2026-08-04 21:23:34.965156749 +0000 UTC m=+2589.717262747"
Aug 04 21:23:37.446914 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:37.446880    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:23:43.766495 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:43.766462    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:43.766889 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:43.766507    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:43.778072 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:43.778047    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:23:44.002796 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:23:44.002767    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:24:16.149359 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:16.149315    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"]
Aug 04 21:24:17.078732 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:17.078697    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"]
Aug 04 21:24:19.652129 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:19.652092    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:24:21.644263 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:21.644229    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"]
Aug 04 21:24:21.644749 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:21.644443    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4" podUID="230dc476-3051-4d22-9f5d-2c360f1a9df0" containerName="payload-processing" containerID="cri-o://122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d" gracePeriod=30
Aug 04 21:24:21.654789 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:21.654759    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"]
Aug 04 21:24:21.655028 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:21.654981    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8" podUID="390a5f44-2fd0-46c5-8a5e-caccce453ed0" containerName="payload-pre-processing" containerID="cri-o://0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4" gracePeriod=30
Aug 04 21:24:21.914027 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:21.913998    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:24:21.939659 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:21.939634    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:24:22.056033 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.055999    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b8zn\" (UniqueName: \"kubernetes.io/projected/390a5f44-2fd0-46c5-8a5e-caccce453ed0-kube-api-access-7b8zn\") pod \"390a5f44-2fd0-46c5-8a5e-caccce453ed0\" (UID: \"390a5f44-2fd0-46c5-8a5e-caccce453ed0\") "
Aug 04 21:24:22.056200 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.056060    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390a5f44-2fd0-46c5-8a5e-caccce453ed0-plugins-config-volume\") pod \"390a5f44-2fd0-46c5-8a5e-caccce453ed0\" (UID: \"390a5f44-2fd0-46c5-8a5e-caccce453ed0\") "
Aug 04 21:24:22.056254 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.056203    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230dc476-3051-4d22-9f5d-2c360f1a9df0-plugins-config-volume\") pod \"230dc476-3051-4d22-9f5d-2c360f1a9df0\" (UID: \"230dc476-3051-4d22-9f5d-2c360f1a9df0\") "
Aug 04 21:24:22.056311 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.056258    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vz29\" (UniqueName: \"kubernetes.io/projected/230dc476-3051-4d22-9f5d-2c360f1a9df0-kube-api-access-7vz29\") pod \"230dc476-3051-4d22-9f5d-2c360f1a9df0\" (UID: \"230dc476-3051-4d22-9f5d-2c360f1a9df0\") "
Aug 04 21:24:22.056434 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.056413    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390a5f44-2fd0-46c5-8a5e-caccce453ed0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "390a5f44-2fd0-46c5-8a5e-caccce453ed0" (UID: "390a5f44-2fd0-46c5-8a5e-caccce453ed0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:24:22.056620 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.056600    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230dc476-3051-4d22-9f5d-2c360f1a9df0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "230dc476-3051-4d22-9f5d-2c360f1a9df0" (UID: "230dc476-3051-4d22-9f5d-2c360f1a9df0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:24:22.056734 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.056693    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390a5f44-2fd0-46c5-8a5e-caccce453ed0-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:22.056734 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.056713    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230dc476-3051-4d22-9f5d-2c360f1a9df0-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:22.058139 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.058114    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390a5f44-2fd0-46c5-8a5e-caccce453ed0-kube-api-access-7b8zn" (OuterVolumeSpecName: "kube-api-access-7b8zn") pod "390a5f44-2fd0-46c5-8a5e-caccce453ed0" (UID: "390a5f44-2fd0-46c5-8a5e-caccce453ed0"). InnerVolumeSpecName "kube-api-access-7b8zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:24:22.058215 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.058159    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230dc476-3051-4d22-9f5d-2c360f1a9df0-kube-api-access-7vz29" (OuterVolumeSpecName: "kube-api-access-7vz29") pod "230dc476-3051-4d22-9f5d-2c360f1a9df0" (UID: "230dc476-3051-4d22-9f5d-2c360f1a9df0"). InnerVolumeSpecName "kube-api-access-7vz29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:24:22.157656 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.157594    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b8zn\" (UniqueName: \"kubernetes.io/projected/390a5f44-2fd0-46c5-8a5e-caccce453ed0-kube-api-access-7b8zn\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:22.157656 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.157617    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vz29\" (UniqueName: \"kubernetes.io/projected/230dc476-3051-4d22-9f5d-2c360f1a9df0-kube-api-access-7vz29\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:22.187205 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.187174    2565 generic.go:356] "Generic (PLEG): container finished" podID="390a5f44-2fd0-46c5-8a5e-caccce453ed0" containerID="0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4" exitCode=0
Aug 04 21:24:22.187376 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.187224    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"
Aug 04 21:24:22.187376 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.187256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8" event={"ID":"390a5f44-2fd0-46c5-8a5e-caccce453ed0","Type":"ContainerDied","Data":"0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4"}
Aug 04 21:24:22.187376 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.187308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8" event={"ID":"390a5f44-2fd0-46c5-8a5e-caccce453ed0","Type":"ContainerDied","Data":"7611f0f8f729349267071911f6c9568ad35dbb8bc91d5d81107d4ab2a990283f"}
Aug 04 21:24:22.187376 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.187324    2565 scope.go:117] "RemoveContainer" containerID="0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4"
Aug 04 21:24:22.188848 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.188824    2565 generic.go:356] "Generic (PLEG): container finished" podID="230dc476-3051-4d22-9f5d-2c360f1a9df0" containerID="122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d" exitCode=0
Aug 04 21:24:22.188964 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.188867    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"
Aug 04 21:24:22.188964 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.188903    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4" event={"ID":"230dc476-3051-4d22-9f5d-2c360f1a9df0","Type":"ContainerDied","Data":"122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d"}
Aug 04 21:24:22.188964 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.188957    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4" event={"ID":"230dc476-3051-4d22-9f5d-2c360f1a9df0","Type":"ContainerDied","Data":"9d6d9b4ec891860917ce24c0d4718c253d48760858718584e38380ae1ccaaee8"}
Aug 04 21:24:22.199483 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.199463    2565 scope.go:117] "RemoveContainer" containerID="0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4"
Aug 04 21:24:22.199774 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:24:22.199749    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4\": container with ID starting with 0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4 not found: ID does not exist" containerID="0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4"
Aug 04 21:24:22.199887 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.199784    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4"} err="failed to get container status \"0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4\": rpc error: code = NotFound desc = could not find container \"0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4\": container with ID starting with 0100ee63ac171afa555f9e07fc4b5916821fa6942b4c38afb65fd9d6bbcf3ed4 not found: ID does not exist"
Aug 04 21:24:22.199887 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.199805    2565 scope.go:117] "RemoveContainer" containerID="122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d"
Aug 04 21:24:22.207651 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.207636    2565 scope.go:117] "RemoveContainer" containerID="122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d"
Aug 04 21:24:22.207882 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:24:22.207865    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d\": container with ID starting with 122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d not found: ID does not exist" containerID="122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d"
Aug 04 21:24:22.207931 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.207890    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d"} err="failed to get container status \"122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d\": rpc error: code = NotFound desc = could not find container \"122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d\": container with ID starting with 122da4f32d7014b33f4fd24d7c19de18a781b581f34ebffc17422ead6c24917d not found: ID does not exist"
Aug 04 21:24:22.211801 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.211778    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"]
Aug 04 21:24:22.214872 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.214851    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-19ad13-55c96fc556-4rgp8"]
Aug 04 21:24:22.224798 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.224778    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"]
Aug 04 21:24:22.228289 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:22.228267    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-19ad13-5697c988b8-sgcn4"]
Aug 04 21:24:23.761039 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:23.761006    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230dc476-3051-4d22-9f5d-2c360f1a9df0" path="/var/lib/kubelet/pods/230dc476-3051-4d22-9f5d-2c360f1a9df0/volumes"
Aug 04 21:24:23.761439 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:23.761383    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="390a5f44-2fd0-46c5-8a5e-caccce453ed0" path="/var/lib/kubelet/pods/390a5f44-2fd0-46c5-8a5e-caccce453ed0/volumes"
Aug 04 21:24:27.067078 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:27.067045    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:24:31.168977 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.168913    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn" podUID="2a266ea3-cea0-43ba-af1b-b2bdc435081d" containerName="main" containerID="cri-o://4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74" gracePeriod=45
Aug 04 21:24:31.418522 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.418497    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:24:31.546549 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.546519    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-model-cache\") pod \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") "
Aug 04 21:24:31.546702 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.546562    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-dshm\") pod \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") "
Aug 04 21:24:31.546702 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.546597    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-home\") pod \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") "
Aug 04 21:24:31.546702 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.546648    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tls-certs\") pod \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") "
Aug 04 21:24:31.546702 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.546687    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tmp-dir\") pod \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") "
Aug 04 21:24:31.546904 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.546712    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vb68\" (UniqueName: \"kubernetes.io/projected/2a266ea3-cea0-43ba-af1b-b2bdc435081d-kube-api-access-5vb68\") pod \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\" (UID: \"2a266ea3-cea0-43ba-af1b-b2bdc435081d\") "
Aug 04 21:24:31.546904 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.546807    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a266ea3-cea0-43ba-af1b-b2bdc435081d" (UID: "2a266ea3-cea0-43ba-af1b-b2bdc435081d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:24:31.546904 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.546857    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-home" (OuterVolumeSpecName: "home") pod "2a266ea3-cea0-43ba-af1b-b2bdc435081d" (UID: "2a266ea3-cea0-43ba-af1b-b2bdc435081d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:24:31.547061 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.547015    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:31.547061 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.547051    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:31.547139 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.547007    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a266ea3-cea0-43ba-af1b-b2bdc435081d" (UID: "2a266ea3-cea0-43ba-af1b-b2bdc435081d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:24:31.548948 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.548929    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-dshm" (OuterVolumeSpecName: "dshm") pod "2a266ea3-cea0-43ba-af1b-b2bdc435081d" (UID: "2a266ea3-cea0-43ba-af1b-b2bdc435081d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:24:31.549023 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.548993    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a266ea3-cea0-43ba-af1b-b2bdc435081d" (UID: "2a266ea3-cea0-43ba-af1b-b2bdc435081d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 21:24:31.549109 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.549034    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a266ea3-cea0-43ba-af1b-b2bdc435081d-kube-api-access-5vb68" (OuterVolumeSpecName: "kube-api-access-5vb68") pod "2a266ea3-cea0-43ba-af1b-b2bdc435081d" (UID: "2a266ea3-cea0-43ba-af1b-b2bdc435081d"). InnerVolumeSpecName "kube-api-access-5vb68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:24:31.648364 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.648320    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:31.648457 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.648367    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:31.648457 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.648382    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a266ea3-cea0-43ba-af1b-b2bdc435081d-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:31.648457 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:31.648394    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vb68\" (UniqueName: \"kubernetes.io/projected/2a266ea3-cea0-43ba-af1b-b2bdc435081d-kube-api-access-5vb68\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:32.098251 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.098191    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv" podUID="6e745330-e6d1-4be8-b4b6-999bb6a2d60f" containerName="main" containerID="cri-o://8f8655671cdabc50d4320c0674191bd316227cd6164bd5becdaa9ded9c2a5aff" gracePeriod=45
Aug 04 21:24:32.240961 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.240930    2565 generic.go:356] "Generic (PLEG): container finished" podID="6e745330-e6d1-4be8-b4b6-999bb6a2d60f" containerID="8f8655671cdabc50d4320c0674191bd316227cd6164bd5becdaa9ded9c2a5aff" exitCode=0
Aug 04 21:24:32.241363 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.241014    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv" event={"ID":"6e745330-e6d1-4be8-b4b6-999bb6a2d60f","Type":"ContainerDied","Data":"8f8655671cdabc50d4320c0674191bd316227cd6164bd5becdaa9ded9c2a5aff"}
Aug 04 21:24:32.242521 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.242495    2565 generic.go:356] "Generic (PLEG): container finished" podID="2a266ea3-cea0-43ba-af1b-b2bdc435081d" containerID="4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74" exitCode=0
Aug 04 21:24:32.242616 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.242530    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"
Aug 04 21:24:32.242616 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.242567    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn" event={"ID":"2a266ea3-cea0-43ba-af1b-b2bdc435081d","Type":"ContainerDied","Data":"4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74"}
Aug 04 21:24:32.242616 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.242600    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn" event={"ID":"2a266ea3-cea0-43ba-af1b-b2bdc435081d","Type":"ContainerDied","Data":"fe7c5fccef1ca229f889fb2c067dd7da58b2ce75d6efaa5671087d428781dd58"}
Aug 04 21:24:32.242729 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.242620    2565 scope.go:117] "RemoveContainer" containerID="4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74"
Aug 04 21:24:32.252414 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.252395    2565 scope.go:117] "RemoveContainer" containerID="4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74"
Aug 04 21:24:32.252692 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:24:32.252671    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74\": container with ID starting with 4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74 not found: ID does not exist" containerID="4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74"
Aug 04 21:24:32.252751 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.252701    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74"} err="failed to get container status \"4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74\": rpc error: code = NotFound desc = could not find container \"4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74\": container with ID starting with 4a017f5f62b64b00df28ede4d1dcffe8430730e6bbfce4319e75aaa9564aac74 not found: ID does not exist"
Aug 04 21:24:32.260977 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.260952    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"]
Aug 04 21:24:32.264544 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.264523    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-19ad13/test-model-19ad13-kserve-757f89987-dsbjn"]
Aug 04 21:24:32.357685 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.357623    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:24:32.455941 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.455904    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tmp-dir\") pod \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") "
Aug 04 21:24:32.456098 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.455969    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6lsm\" (UniqueName: \"kubernetes.io/projected/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-kube-api-access-s6lsm\") pod \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") "
Aug 04 21:24:32.456098 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.455997    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-model-cache\") pod \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") "
Aug 04 21:24:32.456098 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.456035    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-dshm\") pod \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") "
Aug 04 21:24:32.456293 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.456102    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tls-certs\") pod \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") "
Aug 04 21:24:32.456293 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.456138    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-home\") pod \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\" (UID: \"6e745330-e6d1-4be8-b4b6-999bb6a2d60f\") "
Aug 04 21:24:32.456293 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.456208    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e745330-e6d1-4be8-b4b6-999bb6a2d60f" (UID: "6e745330-e6d1-4be8-b4b6-999bb6a2d60f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:24:32.456475 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.456300    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e745330-e6d1-4be8-b4b6-999bb6a2d60f" (UID: "6e745330-e6d1-4be8-b4b6-999bb6a2d60f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:24:32.456475 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.456448    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-home" (OuterVolumeSpecName: "home") pod "6e745330-e6d1-4be8-b4b6-999bb6a2d60f" (UID: "6e745330-e6d1-4be8-b4b6-999bb6a2d60f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:24:32.456566 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.456551    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:32.456618 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.456571    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:32.456618 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.456585    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:32.458353 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.458320    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-dshm" (OuterVolumeSpecName: "dshm") pod "6e745330-e6d1-4be8-b4b6-999bb6a2d60f" (UID: "6e745330-e6d1-4be8-b4b6-999bb6a2d60f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:24:32.458418 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.458335    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-kube-api-access-s6lsm" (OuterVolumeSpecName: "kube-api-access-s6lsm") pod "6e745330-e6d1-4be8-b4b6-999bb6a2d60f" (UID: "6e745330-e6d1-4be8-b4b6-999bb6a2d60f"). InnerVolumeSpecName "kube-api-access-s6lsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:24:32.458461 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.458430    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e745330-e6d1-4be8-b4b6-999bb6a2d60f" (UID: "6e745330-e6d1-4be8-b4b6-999bb6a2d60f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 21:24:32.557310 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.557267    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6lsm\" (UniqueName: \"kubernetes.io/projected/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-kube-api-access-s6lsm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:32.557310 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.557306    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:32.557310 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:32.557320    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e745330-e6d1-4be8-b4b6-999bb6a2d60f-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:33.249537 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.249507    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"
Aug 04 21:24:33.249537 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.249520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv" event={"ID":"6e745330-e6d1-4be8-b4b6-999bb6a2d60f","Type":"ContainerDied","Data":"50d1c4aa049280c2e2bc182054e00233408503999d67f475fa5fde1b803465a0"}
Aug 04 21:24:33.250013 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.249572    2565 scope.go:117] "RemoveContainer" containerID="8f8655671cdabc50d4320c0674191bd316227cd6164bd5becdaa9ded9c2a5aff"
Aug 04 21:24:33.270732 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.270709    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"]
Aug 04 21:24:33.274652 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.274630    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4c3747/test-model-4c3747-kserve-5b77cbb79c-h8bmv"]
Aug 04 21:24:33.760187 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.760153    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a266ea3-cea0-43ba-af1b-b2bdc435081d" path="/var/lib/kubelet/pods/2a266ea3-cea0-43ba-af1b-b2bdc435081d/volumes"
Aug 04 21:24:33.760595 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.760581    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e745330-e6d1-4be8-b4b6-999bb6a2d60f" path="/var/lib/kubelet/pods/6e745330-e6d1-4be8-b4b6-999bb6a2d60f/volumes"
Aug 04 21:24:33.898890 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.898858    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"]
Aug 04 21:24:33.899247 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899235    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a266ea3-cea0-43ba-af1b-b2bdc435081d" containerName="main"
Aug 04 21:24:33.899296 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899249    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a266ea3-cea0-43ba-af1b-b2bdc435081d" containerName="main"
Aug 04 21:24:33.899296 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899261    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="390a5f44-2fd0-46c5-8a5e-caccce453ed0" containerName="payload-pre-processing"
Aug 04 21:24:33.899296 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899267    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="390a5f44-2fd0-46c5-8a5e-caccce453ed0" containerName="payload-pre-processing"
Aug 04 21:24:33.899296 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899278    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e745330-e6d1-4be8-b4b6-999bb6a2d60f" containerName="main"
Aug 04 21:24:33.899296 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899283    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e745330-e6d1-4be8-b4b6-999bb6a2d60f" containerName="main"
Aug 04 21:24:33.899296 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899291    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230dc476-3051-4d22-9f5d-2c360f1a9df0" containerName="payload-processing"
Aug 04 21:24:33.899296 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899296    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="230dc476-3051-4d22-9f5d-2c360f1a9df0" containerName="payload-processing"
Aug 04 21:24:33.899535 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899361    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a266ea3-cea0-43ba-af1b-b2bdc435081d" containerName="main"
Aug 04 21:24:33.899535 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899371    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="230dc476-3051-4d22-9f5d-2c360f1a9df0" containerName="payload-processing"
Aug 04 21:24:33.899535 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899376    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="390a5f44-2fd0-46c5-8a5e-caccce453ed0" containerName="payload-pre-processing"
Aug 04 21:24:33.899535 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.899382    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e745330-e6d1-4be8-b4b6-999bb6a2d60f" containerName="main"
Aug 04 21:24:33.902330 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.902315    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"
Aug 04 21:24:33.908488 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.905635    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-86svs\""
Aug 04 21:24:33.908488 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.905930    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 04 21:24:33.911337 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.911314    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"]
Aug 04 21:24:33.970882 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:33.970849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/796a04c9-3a71-4a96-bc3b-2309f495fdf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk\" (UID: \"796a04c9-3a71-4a96-bc3b-2309f495fdf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"
Aug 04 21:24:34.071581 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:34.071496    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/796a04c9-3a71-4a96-bc3b-2309f495fdf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk\" (UID: \"796a04c9-3a71-4a96-bc3b-2309f495fdf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"
Aug 04 21:24:34.072072 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:34.072052    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/796a04c9-3a71-4a96-bc3b-2309f495fdf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk\" (UID: \"796a04c9-3a71-4a96-bc3b-2309f495fdf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"
Aug 04 21:24:34.217934 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:34.217906    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"
Aug 04 21:24:34.347271 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:34.347183    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"]
Aug 04 21:24:34.349857 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:24:34.349826    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796a04c9_3a71_4a96_bc3b_2309f495fdf1.slice/crio-51723eed2b9a828a0510431e530656a5306c7107cfa0cb7bd38cff2da38c4527 WatchSource:0}: Error finding container 51723eed2b9a828a0510431e530656a5306c7107cfa0cb7bd38cff2da38c4527: Status 404 returned error can't find the container with id 51723eed2b9a828a0510431e530656a5306c7107cfa0cb7bd38cff2da38c4527
Aug 04 21:24:35.262675 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:35.262641    2565 generic.go:356] "Generic (PLEG): container finished" podID="796a04c9-3a71-4a96-bc3b-2309f495fdf1" containerID="18c3ae8d74162515945fb5250a69aa5758db2d6f307441999fda35e34374d699" exitCode=0
Aug 04 21:24:35.262843 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:35.262734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk" event={"ID":"796a04c9-3a71-4a96-bc3b-2309f495fdf1","Type":"ContainerDied","Data":"18c3ae8d74162515945fb5250a69aa5758db2d6f307441999fda35e34374d699"}
Aug 04 21:24:35.262843 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:35.262792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk" event={"ID":"796a04c9-3a71-4a96-bc3b-2309f495fdf1","Type":"ContainerStarted","Data":"51723eed2b9a828a0510431e530656a5306c7107cfa0cb7bd38cff2da38c4527"}
Aug 04 21:24:36.400210 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:36.400184    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"
Aug 04 21:24:36.491871 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:36.491845    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/796a04c9-3a71-4a96-bc3b-2309f495fdf1-maas-api-service-ca\") pod \"796a04c9-3a71-4a96-bc3b-2309f495fdf1\" (UID: \"796a04c9-3a71-4a96-bc3b-2309f495fdf1\") "
Aug 04 21:24:36.492178 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:36.492155    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796a04c9-3a71-4a96-bc3b-2309f495fdf1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "796a04c9-3a71-4a96-bc3b-2309f495fdf1" (UID: "796a04c9-3a71-4a96-bc3b-2309f495fdf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:24:36.593372 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:36.593271    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/796a04c9-3a71-4a96-bc3b-2309f495fdf1-maas-api-service-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:37.272948 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:37.272915    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"
Aug 04 21:24:37.272948 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:37.272941    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk" event={"ID":"796a04c9-3a71-4a96-bc3b-2309f495fdf1","Type":"ContainerDied","Data":"51723eed2b9a828a0510431e530656a5306c7107cfa0cb7bd38cff2da38c4527"}
Aug 04 21:24:37.273144 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:37.272983    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51723eed2b9a828a0510431e530656a5306c7107cfa0cb7bd38cff2da38c4527"
Aug 04 21:24:44.539292 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.539253    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"]
Aug 04 21:24:44.539739 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.539462    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg" podUID="24b49185-1179-40b1-a819-80dc8ebb551d" containerName="payload-processing" containerID="cri-o://6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e" gracePeriod=30
Aug 04 21:24:44.545490 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.545459    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"]
Aug 04 21:24:44.545697 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.545672    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk" podUID="7edf508e-bdc1-4833-9c8f-557160016ead" containerName="payload-pre-processing" containerID="cri-o://8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027" gracePeriod=30
Aug 04 21:24:44.683565 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.683493    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"]
Aug 04 21:24:44.687803 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.687771    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c3747-8crlk"]
Aug 04 21:24:44.798212 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.798144    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:24:44.830114 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.830086    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:24:44.862190 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.862166    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7edf508e-bdc1-4833-9c8f-557160016ead-plugins-config-volume\") pod \"7edf508e-bdc1-4833-9c8f-557160016ead\" (UID: \"7edf508e-bdc1-4833-9c8f-557160016ead\") "
Aug 04 21:24:44.862333 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.862196    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r55zk\" (UniqueName: \"kubernetes.io/projected/24b49185-1179-40b1-a819-80dc8ebb551d-kube-api-access-r55zk\") pod \"24b49185-1179-40b1-a819-80dc8ebb551d\" (UID: \"24b49185-1179-40b1-a819-80dc8ebb551d\") "
Aug 04 21:24:44.862333 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.862237    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2clz\" (UniqueName: \"kubernetes.io/projected/7edf508e-bdc1-4833-9c8f-557160016ead-kube-api-access-b2clz\") pod \"7edf508e-bdc1-4833-9c8f-557160016ead\" (UID: \"7edf508e-bdc1-4833-9c8f-557160016ead\") "
Aug 04 21:24:44.862333 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.862295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b49185-1179-40b1-a819-80dc8ebb551d-plugins-config-volume\") pod \"24b49185-1179-40b1-a819-80dc8ebb551d\" (UID: \"24b49185-1179-40b1-a819-80dc8ebb551d\") "
Aug 04 21:24:44.862714 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.862682    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edf508e-bdc1-4833-9c8f-557160016ead-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7edf508e-bdc1-4833-9c8f-557160016ead" (UID: "7edf508e-bdc1-4833-9c8f-557160016ead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:24:44.862811 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.862752    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b49185-1179-40b1-a819-80dc8ebb551d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24b49185-1179-40b1-a819-80dc8ebb551d" (UID: "24b49185-1179-40b1-a819-80dc8ebb551d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:24:44.864436 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.864408    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7edf508e-bdc1-4833-9c8f-557160016ead-kube-api-access-b2clz" (OuterVolumeSpecName: "kube-api-access-b2clz") pod "7edf508e-bdc1-4833-9c8f-557160016ead" (UID: "7edf508e-bdc1-4833-9c8f-557160016ead"). InnerVolumeSpecName "kube-api-access-b2clz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:24:44.864817 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.864802    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b49185-1179-40b1-a819-80dc8ebb551d-kube-api-access-r55zk" (OuterVolumeSpecName: "kube-api-access-r55zk") pod "24b49185-1179-40b1-a819-80dc8ebb551d" (UID: "24b49185-1179-40b1-a819-80dc8ebb551d"). InnerVolumeSpecName "kube-api-access-r55zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:24:44.963473 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.963444    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b49185-1179-40b1-a819-80dc8ebb551d-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:44.963473 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.963472    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7edf508e-bdc1-4833-9c8f-557160016ead-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:44.963643 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.963481    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r55zk\" (UniqueName: \"kubernetes.io/projected/24b49185-1179-40b1-a819-80dc8ebb551d-kube-api-access-r55zk\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:44.963643 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:44.963490    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2clz\" (UniqueName: \"kubernetes.io/projected/7edf508e-bdc1-4833-9c8f-557160016ead-kube-api-access-b2clz\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:24:45.313719 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.313684    2565 generic.go:356] "Generic (PLEG): container finished" podID="24b49185-1179-40b1-a819-80dc8ebb551d" containerID="6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e" exitCode=0
Aug 04 21:24:45.313878 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.313726    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"
Aug 04 21:24:45.313878 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.313753    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg" event={"ID":"24b49185-1179-40b1-a819-80dc8ebb551d","Type":"ContainerDied","Data":"6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e"}
Aug 04 21:24:45.313878 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.313808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg" event={"ID":"24b49185-1179-40b1-a819-80dc8ebb551d","Type":"ContainerDied","Data":"88cb4a6628cc0ba5af62b0bfdfa7104118900454fb4a590214cdab7fda8a0491"}
Aug 04 21:24:45.313878 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.313842    2565 scope.go:117] "RemoveContainer" containerID="6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e"
Aug 04 21:24:45.315098 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.315073    2565 generic.go:356] "Generic (PLEG): container finished" podID="7edf508e-bdc1-4833-9c8f-557160016ead" containerID="8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027" exitCode=0
Aug 04 21:24:45.315196 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.315112    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"
Aug 04 21:24:45.315196 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.315154    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk" event={"ID":"7edf508e-bdc1-4833-9c8f-557160016ead","Type":"ContainerDied","Data":"8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027"}
Aug 04 21:24:45.315299 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.315207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk" event={"ID":"7edf508e-bdc1-4833-9c8f-557160016ead","Type":"ContainerDied","Data":"cd83bfd0e4caf2f9b0681cfc2b1fbbd7d9db6c42ffc02526d4d845f016386563"}
Aug 04 21:24:45.322844 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.322824    2565 scope.go:117] "RemoveContainer" containerID="6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e"
Aug 04 21:24:45.323123 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:24:45.323072    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e\": container with ID starting with 6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e not found: ID does not exist" containerID="6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e"
Aug 04 21:24:45.323123 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.323094    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e"} err="failed to get container status \"6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e\": rpc error: code = NotFound desc = could not find container \"6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e\": container with ID starting with 6f41b9bd0b153ba544d5640ba042c41c54ec791261f6c26280edc95dd4808d0e not found: ID does not exist"
Aug 04 21:24:45.323123 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.323112    2565 scope.go:117] "RemoveContainer" containerID="8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027"
Aug 04 21:24:45.330934 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.330917    2565 scope.go:117] "RemoveContainer" containerID="8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027"
Aug 04 21:24:45.331182 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:24:45.331161    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027\": container with ID starting with 8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027 not found: ID does not exist" containerID="8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027"
Aug 04 21:24:45.331225 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.331190    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027"} err="failed to get container status \"8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027\": rpc error: code = NotFound desc = could not find container \"8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027\": container with ID starting with 8c17a6a560024b2e0dde0e9f2855f7cb9087afdb9e86752e911a808d20ac0027 not found: ID does not exist"
Aug 04 21:24:45.350893 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.350870    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"]
Aug 04 21:24:45.362726 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.362702    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c3747-86c79bcc8d-v4rrk"]
Aug 04 21:24:45.379824 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.379801    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"]
Aug 04 21:24:45.385506 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.385485    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c3747-7457dbfcd4-gf7zg"]
Aug 04 21:24:45.760169 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.760127    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24b49185-1179-40b1-a819-80dc8ebb551d" path="/var/lib/kubelet/pods/24b49185-1179-40b1-a819-80dc8ebb551d/volumes"
Aug 04 21:24:45.760523 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.760483    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="796a04c9-3a71-4a96-bc3b-2309f495fdf1" path="/var/lib/kubelet/pods/796a04c9-3a71-4a96-bc3b-2309f495fdf1/volumes"
Aug 04 21:24:45.760803 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:45.760790    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7edf508e-bdc1-4833-9c8f-557160016ead" path="/var/lib/kubelet/pods/7edf508e-bdc1-4833-9c8f-557160016ead/volumes"
Aug 04 21:24:46.668235 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:46.668202    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:24:57.074254 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.074216    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"]
Aug 04 21:24:57.074731 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.074628    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b49185-1179-40b1-a819-80dc8ebb551d" containerName="payload-processing"
Aug 04 21:24:57.074731 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.074644    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b49185-1179-40b1-a819-80dc8ebb551d" containerName="payload-processing"
Aug 04 21:24:57.074731 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.074655    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="796a04c9-3a71-4a96-bc3b-2309f495fdf1" containerName="revoke-keys"
Aug 04 21:24:57.074731 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.074660    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="796a04c9-3a71-4a96-bc3b-2309f495fdf1" containerName="revoke-keys"
Aug 04 21:24:57.074731 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.074685    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7edf508e-bdc1-4833-9c8f-557160016ead" containerName="payload-pre-processing"
Aug 04 21:24:57.074731 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.074691    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7edf508e-bdc1-4833-9c8f-557160016ead" containerName="payload-pre-processing"
Aug 04 21:24:57.074921 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.074754    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7edf508e-bdc1-4833-9c8f-557160016ead" containerName="payload-pre-processing"
Aug 04 21:24:57.074921 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.074761    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="24b49185-1179-40b1-a819-80dc8ebb551d" containerName="payload-processing"
Aug 04 21:24:57.074921 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.074769    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="796a04c9-3a71-4a96-bc3b-2309f495fdf1" containerName="revoke-keys"
Aug 04 21:24:57.079222 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.079204    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:24:57.081882 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.081854    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a8febf\""
Aug 04 21:24:57.094066 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.094036    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"]
Aug 04 21:24:57.097304 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.097188    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"]
Aug 04 21:24:57.102002 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.101980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:24:57.104448 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.104426    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a8febf-dockercfg-l8fmp\""
Aug 04 21:24:57.112319 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.112294    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"]
Aug 04 21:24:57.184909 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.184875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsqvb\" (UniqueName: \"kubernetes.io/projected/6a860ba7-034f-47df-b568-01bfa3301b62-kube-api-access-rsqvb\") pod \"payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm\" (UID: \"6a860ba7-034f-47df-b568-01bfa3301b62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:24:57.185073 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.184942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57\" (UID: \"eb31ab1a-5d59-4c51-a927-54b9e262c2b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:24:57.185073 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.185010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a860ba7-034f-47df-b568-01bfa3301b62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm\" (UID: \"6a860ba7-034f-47df-b568-01bfa3301b62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:24:57.185073 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.185052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn842\" (UniqueName: \"kubernetes.io/projected/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-kube-api-access-jn842\") pod \"payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57\" (UID: \"eb31ab1a-5d59-4c51-a927-54b9e262c2b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:24:57.285585 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.285549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsqvb\" (UniqueName: \"kubernetes.io/projected/6a860ba7-034f-47df-b568-01bfa3301b62-kube-api-access-rsqvb\") pod \"payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm\" (UID: \"6a860ba7-034f-47df-b568-01bfa3301b62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:24:57.285777 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.285614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57\" (UID: \"eb31ab1a-5d59-4c51-a927-54b9e262c2b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:24:57.285777 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.285651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a860ba7-034f-47df-b568-01bfa3301b62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm\" (UID: \"6a860ba7-034f-47df-b568-01bfa3301b62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:24:57.285777 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.285675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn842\" (UniqueName: \"kubernetes.io/projected/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-kube-api-access-jn842\") pod \"payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57\" (UID: \"eb31ab1a-5d59-4c51-a927-54b9e262c2b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:24:57.286443 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.286422    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57\" (UID: \"eb31ab1a-5d59-4c51-a927-54b9e262c2b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:24:57.286504 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.286442    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a860ba7-034f-47df-b568-01bfa3301b62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm\" (UID: \"6a860ba7-034f-47df-b568-01bfa3301b62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:24:57.294174 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.294139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn842\" (UniqueName: \"kubernetes.io/projected/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-kube-api-access-jn842\") pod \"payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57\" (UID: \"eb31ab1a-5d59-4c51-a927-54b9e262c2b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:24:57.294312 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.294148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsqvb\" (UniqueName: \"kubernetes.io/projected/6a860ba7-034f-47df-b568-01bfa3301b62-kube-api-access-rsqvb\") pod \"payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm\" (UID: \"6a860ba7-034f-47df-b568-01bfa3301b62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:24:57.394825 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.394735    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:24:57.414831 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.414802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:24:57.752088 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.752053    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"]
Aug 04 21:24:57.756576 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:24:57.756547    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a860ba7_034f_47df_b568_01bfa3301b62.slice/crio-c88ccf88372fafb33c1bc07f90a29e917cd1c32d98137bfb8e775678c0a5d122 WatchSource:0}: Error finding container c88ccf88372fafb33c1bc07f90a29e917cd1c32d98137bfb8e775678c0a5d122: Status 404 returned error can't find the container with id c88ccf88372fafb33c1bc07f90a29e917cd1c32d98137bfb8e775678c0a5d122
Aug 04 21:24:57.779894 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:57.779870    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"]
Aug 04 21:24:57.790608 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:24:57.790574    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb31ab1a_5d59_4c51_a927_54b9e262c2b6.slice/crio-22f68fac41def20538da728baffe7b8895728449e5a83256a13da8e760f23370 WatchSource:0}: Error finding container 22f68fac41def20538da728baffe7b8895728449e5a83256a13da8e760f23370: Status 404 returned error can't find the container with id 22f68fac41def20538da728baffe7b8895728449e5a83256a13da8e760f23370
Aug 04 21:24:58.382457 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:58.382423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57" event={"ID":"eb31ab1a-5d59-4c51-a927-54b9e262c2b6","Type":"ContainerStarted","Data":"97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b"}
Aug 04 21:24:58.382863 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:58.382477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57" event={"ID":"eb31ab1a-5d59-4c51-a927-54b9e262c2b6","Type":"ContainerStarted","Data":"22f68fac41def20538da728baffe7b8895728449e5a83256a13da8e760f23370"}
Aug 04 21:24:58.382863 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:58.382535    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:24:58.383916 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:58.383894    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm" event={"ID":"6a860ba7-034f-47df-b568-01bfa3301b62","Type":"ContainerStarted","Data":"30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82"}
Aug 04 21:24:58.384017 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:58.383942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm" event={"ID":"6a860ba7-034f-47df-b568-01bfa3301b62","Type":"ContainerStarted","Data":"c88ccf88372fafb33c1bc07f90a29e917cd1c32d98137bfb8e775678c0a5d122"}
Aug 04 21:24:58.384017 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:58.383965    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:24:58.399759 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:58.399712    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57" podStartSLOduration=1.399699386 podStartE2EDuration="1.399699386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:24:58.399146892 +0000 UTC m=+2673.151252870" watchObservedRunningTime="2026-08-04 21:24:58.399699386 +0000 UTC m=+2673.151805363"
Aug 04 21:24:58.415295 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:58.415253    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm" podStartSLOduration=1.415238492 podStartE2EDuration="1.415238492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:24:58.414228746 +0000 UTC m=+2673.166334727" watchObservedRunningTime="2026-08-04 21:24:58.415238492 +0000 UTC m=+2673.167344469"
Aug 04 21:24:59.051110 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:24:59.051071    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:25:09.390457 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:09.390406    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm" podUID="6a860ba7-034f-47df-b568-01bfa3301b62" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.58:9004: connect: connection refused"
Aug 04 21:25:09.390969 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:09.390951    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:25:19.389081 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:19.389009    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:25:25.926100 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:25.926065    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:25:25.929771 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:25.929746    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:25:26.234104 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.234073    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"]
Aug 04 21:25:26.238871 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.238855    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:25:26.241618 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.241597    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c4ea15-dockercfg-5h2pm\""
Aug 04 21:25:26.241739 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.241674    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c4ea15\""
Aug 04 21:25:26.248002 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.247981    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"]
Aug 04 21:25:26.260706 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.260684    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"]
Aug 04 21:25:26.264203 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.264185    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:25:26.272757 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.272740    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"]
Aug 04 21:25:26.342385 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.342331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d257c394-b3e4-4104-9a4c-1a18decd7c1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf\" (UID: \"d257c394-b3e4-4104-9a4c-1a18decd7c1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:25:26.342537 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.342504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwds9\" (UniqueName: \"kubernetes.io/projected/9fb0718a-73c9-4135-860f-c7fb09628239-kube-api-access-kwds9\") pod \"payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz\" (UID: \"9fb0718a-73c9-4135-860f-c7fb09628239\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:25:26.342599 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.342538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lpmg\" (UniqueName: \"kubernetes.io/projected/d257c394-b3e4-4104-9a4c-1a18decd7c1f-kube-api-access-7lpmg\") pod \"payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf\" (UID: \"d257c394-b3e4-4104-9a4c-1a18decd7c1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:25:26.342599 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.342571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb0718a-73c9-4135-860f-c7fb09628239-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz\" (UID: \"9fb0718a-73c9-4135-860f-c7fb09628239\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:25:26.443732 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.443703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d257c394-b3e4-4104-9a4c-1a18decd7c1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf\" (UID: \"d257c394-b3e4-4104-9a4c-1a18decd7c1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:25:26.443855 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.443800    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwds9\" (UniqueName: \"kubernetes.io/projected/9fb0718a-73c9-4135-860f-c7fb09628239-kube-api-access-kwds9\") pod \"payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz\" (UID: \"9fb0718a-73c9-4135-860f-c7fb09628239\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:25:26.443855 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.443828    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpmg\" (UniqueName: \"kubernetes.io/projected/d257c394-b3e4-4104-9a4c-1a18decd7c1f-kube-api-access-7lpmg\") pod \"payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf\" (UID: \"d257c394-b3e4-4104-9a4c-1a18decd7c1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:25:26.443938 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.443858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb0718a-73c9-4135-860f-c7fb09628239-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz\" (UID: \"9fb0718a-73c9-4135-860f-c7fb09628239\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:25:26.444654 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.444632    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d257c394-b3e4-4104-9a4c-1a18decd7c1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf\" (UID: \"d257c394-b3e4-4104-9a4c-1a18decd7c1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:25:26.444716 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.444637    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb0718a-73c9-4135-860f-c7fb09628239-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz\" (UID: \"9fb0718a-73c9-4135-860f-c7fb09628239\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:25:26.452655 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.452634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lpmg\" (UniqueName: \"kubernetes.io/projected/d257c394-b3e4-4104-9a4c-1a18decd7c1f-kube-api-access-7lpmg\") pod \"payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf\" (UID: \"d257c394-b3e4-4104-9a4c-1a18decd7c1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:25:26.452748 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.452656    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwds9\" (UniqueName: \"kubernetes.io/projected/9fb0718a-73c9-4135-860f-c7fb09628239-kube-api-access-kwds9\") pod \"payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz\" (UID: \"9fb0718a-73c9-4135-860f-c7fb09628239\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:25:26.550942 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.550866    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:25:26.575022 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.574993    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:25:26.693877 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.693855    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"]
Aug 04 21:25:26.697271 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:25:26.697239    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb0718a_73c9_4135_860f_c7fb09628239.slice/crio-94d425c7a4f7db1cf85cc8dfe6ed88ea8bcbd0ad96c9e999382ff9293bc7cb75 WatchSource:0}: Error finding container 94d425c7a4f7db1cf85cc8dfe6ed88ea8bcbd0ad96c9e999382ff9293bc7cb75: Status 404 returned error can't find the container with id 94d425c7a4f7db1cf85cc8dfe6ed88ea8bcbd0ad96c9e999382ff9293bc7cb75
Aug 04 21:25:26.721686 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:26.721665    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"]
Aug 04 21:25:26.729448 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:25:26.729426    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd257c394_b3e4_4104_9a4c_1a18decd7c1f.slice/crio-23acc4253653c0a3f499234510b2d58f52d22114c22b96f5c9d181e139ab314c WatchSource:0}: Error finding container 23acc4253653c0a3f499234510b2d58f52d22114c22b96f5c9d181e139ab314c: Status 404 returned error can't find the container with id 23acc4253653c0a3f499234510b2d58f52d22114c22b96f5c9d181e139ab314c
Aug 04 21:25:27.532092 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:27.532058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf" event={"ID":"d257c394-b3e4-4104-9a4c-1a18decd7c1f","Type":"ContainerStarted","Data":"b188ae9788667164ecc23f5634ecde288ba3413bc3530e5fed0634a7786d9c00"}
Aug 04 21:25:27.532541 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:27.532116    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf" event={"ID":"d257c394-b3e4-4104-9a4c-1a18decd7c1f","Type":"ContainerStarted","Data":"23acc4253653c0a3f499234510b2d58f52d22114c22b96f5c9d181e139ab314c"}
Aug 04 21:25:27.532541 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:27.532218    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:25:27.533667 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:27.533641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz" event={"ID":"9fb0718a-73c9-4135-860f-c7fb09628239","Type":"ContainerStarted","Data":"7ccfb77a65c02f55b5628c1521557252de5fc7068f14e7752528b60cce923537"}
Aug 04 21:25:27.533788 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:27.533685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz" event={"ID":"9fb0718a-73c9-4135-860f-c7fb09628239","Type":"ContainerStarted","Data":"94d425c7a4f7db1cf85cc8dfe6ed88ea8bcbd0ad96c9e999382ff9293bc7cb75"}
Aug 04 21:25:27.533788 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:27.533714    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:25:27.552185 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:27.552125    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf" podStartSLOduration=1.5521075770000001 podStartE2EDuration="1.552107577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:25:27.551286617 +0000 UTC m=+2702.303392595" watchObservedRunningTime="2026-08-04 21:25:27.552107577 +0000 UTC m=+2702.304213555"
Aug 04 21:25:27.571636 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:27.571582    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz" podStartSLOduration=1.5715631509999999 podStartE2EDuration="1.571563151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:25:27.570945306 +0000 UTC m=+2702.323051282" watchObservedRunningTime="2026-08-04 21:25:27.571563151 +0000 UTC m=+2702.323669123"
Aug 04 21:25:28.444375 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:28.444326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:25:38.541234 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:38.541196    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:25:38.541747 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:38.541285    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:25:52.267616 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:52.267581    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"]
Aug 04 21:25:52.271151 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:52.271136    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"
Aug 04 21:25:52.273534 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:52.273499    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-86svs\""
Aug 04 21:25:52.273534 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:52.273514    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 04 21:25:52.279212 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:52.279104    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"]
Aug 04 21:25:52.373787 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:52.373752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b2f57f-4632-4dbb-b801-6d5910c1e134-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8\" (UID: \"c4b2f57f-4632-4dbb-b801-6d5910c1e134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"
Aug 04 21:25:52.474705 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:52.474663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b2f57f-4632-4dbb-b801-6d5910c1e134-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8\" (UID: \"c4b2f57f-4632-4dbb-b801-6d5910c1e134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"
Aug 04 21:25:52.475295 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:52.475275    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b2f57f-4632-4dbb-b801-6d5910c1e134-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8\" (UID: \"c4b2f57f-4632-4dbb-b801-6d5910c1e134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"
Aug 04 21:25:52.583404 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:52.583325    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"
Aug 04 21:25:52.719960 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:52.719933    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"]
Aug 04 21:25:52.721598 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:25:52.721570    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b2f57f_4632_4dbb_b801_6d5910c1e134.slice/crio-f9a3dec2f1c46ef6d51e26de9710f318cbdff406a214ceeae00e2e59d3c0efb9 WatchSource:0}: Error finding container f9a3dec2f1c46ef6d51e26de9710f318cbdff406a214ceeae00e2e59d3c0efb9: Status 404 returned error can't find the container with id f9a3dec2f1c46ef6d51e26de9710f318cbdff406a214ceeae00e2e59d3c0efb9
Aug 04 21:25:53.433362 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:25:53.433162    2565 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b2f57f_4632_4dbb_b801_6d5910c1e134.slice/crio-5902bd6422c854c388a929a7959c221ae4200165eafcf800a474a67134581545.scope/memory.min": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b2f57f_4632_4dbb_b801_6d5910c1e134.slice/crio-5902bd6422c854c388a929a7959c221ae4200165eafcf800a474a67134581545.scope/memory.min: no such device
Aug 04 21:25:53.675009 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:53.674980    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4b2f57f-4632-4dbb-b801-6d5910c1e134" containerID="5902bd6422c854c388a929a7959c221ae4200165eafcf800a474a67134581545" exitCode=0
Aug 04 21:25:53.675174 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:53.675057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8" event={"ID":"c4b2f57f-4632-4dbb-b801-6d5910c1e134","Type":"ContainerDied","Data":"5902bd6422c854c388a929a7959c221ae4200165eafcf800a474a67134581545"}
Aug 04 21:25:53.675174 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:53.675100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8" event={"ID":"c4b2f57f-4632-4dbb-b801-6d5910c1e134","Type":"ContainerStarted","Data":"f9a3dec2f1c46ef6d51e26de9710f318cbdff406a214ceeae00e2e59d3c0efb9"}
Aug 04 21:25:54.814673 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:54.814647    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"
Aug 04 21:25:54.894057 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:54.894027    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b2f57f-4632-4dbb-b801-6d5910c1e134-maas-api-service-ca\") pod \"c4b2f57f-4632-4dbb-b801-6d5910c1e134\" (UID: \"c4b2f57f-4632-4dbb-b801-6d5910c1e134\") "
Aug 04 21:25:54.894385 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:54.894364    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b2f57f-4632-4dbb-b801-6d5910c1e134-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4b2f57f-4632-4dbb-b801-6d5910c1e134" (UID: "c4b2f57f-4632-4dbb-b801-6d5910c1e134"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:25:54.995025 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:54.994992    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b2f57f-4632-4dbb-b801-6d5910c1e134-maas-api-service-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:25:55.686290 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:55.686260    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"
Aug 04 21:25:55.686470 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:55.686300    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8" event={"ID":"c4b2f57f-4632-4dbb-b801-6d5910c1e134","Type":"ContainerDied","Data":"f9a3dec2f1c46ef6d51e26de9710f318cbdff406a214ceeae00e2e59d3c0efb9"}
Aug 04 21:25:55.686470 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:25:55.686389    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9a3dec2f1c46ef6d51e26de9710f318cbdff406a214ceeae00e2e59d3c0efb9"
Aug 04 21:26:02.119897 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.119796    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"]
Aug 04 21:26:02.120527 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.120392    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57" podUID="eb31ab1a-5d59-4c51-a927-54b9e262c2b6" containerName="payload-processing" containerID="cri-o://97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b" gracePeriod=30
Aug 04 21:26:02.123327 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.123298    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"]
Aug 04 21:26:02.124878 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.124817    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm" podUID="6a860ba7-034f-47df-b568-01bfa3301b62" containerName="payload-pre-processing" containerID="cri-o://30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82" gracePeriod=30
Aug 04 21:26:02.247334 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.247302    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"]
Aug 04 21:26:02.255483 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.255446    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8febf-v2wz8"]
Aug 04 21:26:02.376845 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.376783    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:26:02.419427 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.419401    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:26:02.467577 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.467548    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn842\" (UniqueName: \"kubernetes.io/projected/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-kube-api-access-jn842\") pod \"eb31ab1a-5d59-4c51-a927-54b9e262c2b6\" (UID: \"eb31ab1a-5d59-4c51-a927-54b9e262c2b6\") "
Aug 04 21:26:02.467765 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.467592    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsqvb\" (UniqueName: \"kubernetes.io/projected/6a860ba7-034f-47df-b568-01bfa3301b62-kube-api-access-rsqvb\") pod \"6a860ba7-034f-47df-b568-01bfa3301b62\" (UID: \"6a860ba7-034f-47df-b568-01bfa3301b62\") "
Aug 04 21:26:02.467765 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.467638    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a860ba7-034f-47df-b568-01bfa3301b62-plugins-config-volume\") pod \"6a860ba7-034f-47df-b568-01bfa3301b62\" (UID: \"6a860ba7-034f-47df-b568-01bfa3301b62\") "
Aug 04 21:26:02.467765 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.467682    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-plugins-config-volume\") pod \"eb31ab1a-5d59-4c51-a927-54b9e262c2b6\" (UID: \"eb31ab1a-5d59-4c51-a927-54b9e262c2b6\") "
Aug 04 21:26:02.468065 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.468040    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a860ba7-034f-47df-b568-01bfa3301b62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a860ba7-034f-47df-b568-01bfa3301b62" (UID: "6a860ba7-034f-47df-b568-01bfa3301b62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:26:02.468137 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.468070    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb31ab1a-5d59-4c51-a927-54b9e262c2b6" (UID: "eb31ab1a-5d59-4c51-a927-54b9e262c2b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:26:02.469788 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.469768    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-kube-api-access-jn842" (OuterVolumeSpecName: "kube-api-access-jn842") pod "eb31ab1a-5d59-4c51-a927-54b9e262c2b6" (UID: "eb31ab1a-5d59-4c51-a927-54b9e262c2b6"). InnerVolumeSpecName "kube-api-access-jn842". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:26:02.469858 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.469816    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a860ba7-034f-47df-b568-01bfa3301b62-kube-api-access-rsqvb" (OuterVolumeSpecName: "kube-api-access-rsqvb") pod "6a860ba7-034f-47df-b568-01bfa3301b62" (UID: "6a860ba7-034f-47df-b568-01bfa3301b62"). InnerVolumeSpecName "kube-api-access-rsqvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:26:02.568421 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.568384    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsqvb\" (UniqueName: \"kubernetes.io/projected/6a860ba7-034f-47df-b568-01bfa3301b62-kube-api-access-rsqvb\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:26:02.568421 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.568414    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a860ba7-034f-47df-b568-01bfa3301b62-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:26:02.568421 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.568424    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:26:02.568421 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.568433    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn842\" (UniqueName: \"kubernetes.io/projected/eb31ab1a-5d59-4c51-a927-54b9e262c2b6-kube-api-access-jn842\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:26:02.722292 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.722253    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb31ab1a-5d59-4c51-a927-54b9e262c2b6" containerID="97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b" exitCode=0
Aug 04 21:26:02.722503 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.722297    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"
Aug 04 21:26:02.722503 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.722337    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57" event={"ID":"eb31ab1a-5d59-4c51-a927-54b9e262c2b6","Type":"ContainerDied","Data":"97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b"}
Aug 04 21:26:02.722503 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.722405    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57" event={"ID":"eb31ab1a-5d59-4c51-a927-54b9e262c2b6","Type":"ContainerDied","Data":"22f68fac41def20538da728baffe7b8895728449e5a83256a13da8e760f23370"}
Aug 04 21:26:02.722503 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.722435    2565 scope.go:117] "RemoveContainer" containerID="97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b"
Aug 04 21:26:02.723731 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.723707    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a860ba7-034f-47df-b568-01bfa3301b62" containerID="30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82" exitCode=0
Aug 04 21:26:02.723840 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.723732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm" event={"ID":"6a860ba7-034f-47df-b568-01bfa3301b62","Type":"ContainerDied","Data":"30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82"}
Aug 04 21:26:02.723840 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.723745    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"
Aug 04 21:26:02.723840 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.723788    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm" event={"ID":"6a860ba7-034f-47df-b568-01bfa3301b62","Type":"ContainerDied","Data":"c88ccf88372fafb33c1bc07f90a29e917cd1c32d98137bfb8e775678c0a5d122"}
Aug 04 21:26:02.731427 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.731402    2565 scope.go:117] "RemoveContainer" containerID="97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b"
Aug 04 21:26:02.731845 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:26:02.731678    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b\": container with ID starting with 97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b not found: ID does not exist" containerID="97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b"
Aug 04 21:26:02.731845 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.731710    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b"} err="failed to get container status \"97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b\": rpc error: code = NotFound desc = could not find container \"97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b\": container with ID starting with 97d9c0d4ef00155ae59c3d834007ff43393e6f398866234fb5c2b3cc4e57e06b not found: ID does not exist"
Aug 04 21:26:02.731845 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.731731    2565 scope.go:117] "RemoveContainer" containerID="30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82"
Aug 04 21:26:02.739599 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.739586    2565 scope.go:117] "RemoveContainer" containerID="30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82"
Aug 04 21:26:02.739862 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:26:02.739845    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82\": container with ID starting with 30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82 not found: ID does not exist" containerID="30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82"
Aug 04 21:26:02.739908 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.739870    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82"} err="failed to get container status \"30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82\": rpc error: code = NotFound desc = could not find container \"30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82\": container with ID starting with 30357eb5d3609be7b3aa3048b316dc24b4e75c5ee3f7806ab33bb75e26495d82 not found: ID does not exist"
Aug 04 21:26:02.747941 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.747918    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"]
Aug 04 21:26:02.751597 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.751578    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8febf-64bc54fff6-wvh57"]
Aug 04 21:26:02.761869 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.761846    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"]
Aug 04 21:26:02.768098 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:02.768078    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8febf-5dbd69494d-lvlrm"]
Aug 04 21:26:03.760932 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:03.760896    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a860ba7-034f-47df-b568-01bfa3301b62" path="/var/lib/kubelet/pods/6a860ba7-034f-47df-b568-01bfa3301b62/volumes"
Aug 04 21:26:03.761313 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:03.761278    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4b2f57f-4632-4dbb-b801-6d5910c1e134" path="/var/lib/kubelet/pods/c4b2f57f-4632-4dbb-b801-6d5910c1e134/volumes"
Aug 04 21:26:03.761685 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:03.761670    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb31ab1a-5d59-4c51-a927-54b9e262c2b6" path="/var/lib/kubelet/pods/eb31ab1a-5d59-4c51-a927-54b9e262c2b6/volumes"
Aug 04 21:26:04.349032 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:04.348998    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:26:09.069619 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.069573    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"]
Aug 04 21:26:09.070019 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.069959    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a860ba7-034f-47df-b568-01bfa3301b62" containerName="payload-pre-processing"
Aug 04 21:26:09.070019 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.069970    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a860ba7-034f-47df-b568-01bfa3301b62" containerName="payload-pre-processing"
Aug 04 21:26:09.070019 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.069998    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb31ab1a-5d59-4c51-a927-54b9e262c2b6" containerName="payload-processing"
Aug 04 21:26:09.070019 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.070003    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb31ab1a-5d59-4c51-a927-54b9e262c2b6" containerName="payload-processing"
Aug 04 21:26:09.070019 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.070014    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b2f57f-4632-4dbb-b801-6d5910c1e134" containerName="revoke-keys"
Aug 04 21:26:09.070019 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.070019    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b2f57f-4632-4dbb-b801-6d5910c1e134" containerName="revoke-keys"
Aug 04 21:26:09.070213 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.070072    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb31ab1a-5d59-4c51-a927-54b9e262c2b6" containerName="payload-processing"
Aug 04 21:26:09.070213 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.070082    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a860ba7-034f-47df-b568-01bfa3301b62" containerName="payload-pre-processing"
Aug 04 21:26:09.070213 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.070088    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4b2f57f-4632-4dbb-b801-6d5910c1e134" containerName="revoke-keys"
Aug 04 21:26:09.073139 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.073123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"
Aug 04 21:26:09.075828 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.075804    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 04 21:26:09.075928 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.075804    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-86svs\""
Aug 04 21:26:09.081321 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.081300    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"]
Aug 04 21:26:09.120838 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.120814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295b49cb-dfa2-4c01-bc2f-b10605bcdf9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z\" (UID: \"295b49cb-dfa2-4c01-bc2f-b10605bcdf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"
Aug 04 21:26:09.222174 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.222141    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295b49cb-dfa2-4c01-bc2f-b10605bcdf9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z\" (UID: \"295b49cb-dfa2-4c01-bc2f-b10605bcdf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"
Aug 04 21:26:09.222832 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.222808    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295b49cb-dfa2-4c01-bc2f-b10605bcdf9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z\" (UID: \"295b49cb-dfa2-4c01-bc2f-b10605bcdf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"
Aug 04 21:26:09.384243 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.384138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"
Aug 04 21:26:09.516368 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.516322    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"]
Aug 04 21:26:09.519467 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:26:09.519437    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295b49cb_dfa2_4c01_bc2f_b10605bcdf9c.slice/crio-8eb73995cb3b0c9aa16f117cedc35c1bb4fd5c3c434197a0b50a9579ed6ffec0 WatchSource:0}: Error finding container 8eb73995cb3b0c9aa16f117cedc35c1bb4fd5c3c434197a0b50a9579ed6ffec0: Status 404 returned error can't find the container with id 8eb73995cb3b0c9aa16f117cedc35c1bb4fd5c3c434197a0b50a9579ed6ffec0
Aug 04 21:26:09.761857 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:09.761822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z" event={"ID":"295b49cb-dfa2-4c01-bc2f-b10605bcdf9c","Type":"ContainerStarted","Data":"8eb73995cb3b0c9aa16f117cedc35c1bb4fd5c3c434197a0b50a9579ed6ffec0"}
Aug 04 21:26:10.768631 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:10.768601    2565 generic.go:356] "Generic (PLEG): container finished" podID="295b49cb-dfa2-4c01-bc2f-b10605bcdf9c" containerID="309c59e77bc5751f311a9edb1d547b4517a19075889043453fe39c8e0c134964" exitCode=0
Aug 04 21:26:10.768978 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:10.768680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z" event={"ID":"295b49cb-dfa2-4c01-bc2f-b10605bcdf9c","Type":"ContainerDied","Data":"309c59e77bc5751f311a9edb1d547b4517a19075889043453fe39c8e0c134964"}
Aug 04 21:26:11.910761 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:11.910739    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"
Aug 04 21:26:11.951313 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:11.951280    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295b49cb-dfa2-4c01-bc2f-b10605bcdf9c-maas-api-service-ca\") pod \"295b49cb-dfa2-4c01-bc2f-b10605bcdf9c\" (UID: \"295b49cb-dfa2-4c01-bc2f-b10605bcdf9c\") "
Aug 04 21:26:11.951620 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:11.951597    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295b49cb-dfa2-4c01-bc2f-b10605bcdf9c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "295b49cb-dfa2-4c01-bc2f-b10605bcdf9c" (UID: "295b49cb-dfa2-4c01-bc2f-b10605bcdf9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:26:12.052622 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:12.052537    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295b49cb-dfa2-4c01-bc2f-b10605bcdf9c-maas-api-service-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:26:12.779683 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:12.779654    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"
Aug 04 21:26:12.779883 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:12.779695    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z" event={"ID":"295b49cb-dfa2-4c01-bc2f-b10605bcdf9c","Type":"ContainerDied","Data":"8eb73995cb3b0c9aa16f117cedc35c1bb4fd5c3c434197a0b50a9579ed6ffec0"}
Aug 04 21:26:12.779883 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:12.779743    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eb73995cb3b0c9aa16f117cedc35c1bb4fd5c3c434197a0b50a9579ed6ffec0"
Aug 04 21:26:24.645392 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.645333    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"]
Aug 04 21:26:24.645807 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.645554    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf" podUID="d257c394-b3e4-4104-9a4c-1a18decd7c1f" containerName="payload-processing" containerID="cri-o://b188ae9788667164ecc23f5634ecde288ba3413bc3530e5fed0634a7786d9c00" gracePeriod=30
Aug 04 21:26:24.654476 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.654443    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"]
Aug 04 21:26:24.655688 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.655005    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz" podUID="9fb0718a-73c9-4135-860f-c7fb09628239" containerName="payload-pre-processing" containerID="cri-o://7ccfb77a65c02f55b5628c1521557252de5fc7068f14e7752528b60cce923537" gracePeriod=30
Aug 04 21:26:24.786635 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.786594    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"]
Aug 04 21:26:24.788414 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.788384    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4ea15-k9g6z"]
Aug 04 21:26:24.841261 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.841234    2565 generic.go:356] "Generic (PLEG): container finished" podID="d257c394-b3e4-4104-9a4c-1a18decd7c1f" containerID="b188ae9788667164ecc23f5634ecde288ba3413bc3530e5fed0634a7786d9c00" exitCode=0
Aug 04 21:26:24.841407 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.841308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf" event={"ID":"d257c394-b3e4-4104-9a4c-1a18decd7c1f","Type":"ContainerDied","Data":"b188ae9788667164ecc23f5634ecde288ba3413bc3530e5fed0634a7786d9c00"}
Aug 04 21:26:24.842891 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.842868    2565 generic.go:356] "Generic (PLEG): container finished" podID="9fb0718a-73c9-4135-860f-c7fb09628239" containerID="7ccfb77a65c02f55b5628c1521557252de5fc7068f14e7752528b60cce923537" exitCode=0
Aug 04 21:26:24.843020 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.842912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz" event={"ID":"9fb0718a-73c9-4135-860f-c7fb09628239","Type":"ContainerDied","Data":"7ccfb77a65c02f55b5628c1521557252de5fc7068f14e7752528b60cce923537"}
Aug 04 21:26:24.913533 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.913511    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:26:24.933065 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.933044    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:26:24.972718 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.972692    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d257c394-b3e4-4104-9a4c-1a18decd7c1f-plugins-config-volume\") pod \"d257c394-b3e4-4104-9a4c-1a18decd7c1f\" (UID: \"d257c394-b3e4-4104-9a4c-1a18decd7c1f\") "
Aug 04 21:26:24.972857 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.972778    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lpmg\" (UniqueName: \"kubernetes.io/projected/d257c394-b3e4-4104-9a4c-1a18decd7c1f-kube-api-access-7lpmg\") pod \"d257c394-b3e4-4104-9a4c-1a18decd7c1f\" (UID: \"d257c394-b3e4-4104-9a4c-1a18decd7c1f\") "
Aug 04 21:26:24.973051 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.973027    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d257c394-b3e4-4104-9a4c-1a18decd7c1f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d257c394-b3e4-4104-9a4c-1a18decd7c1f" (UID: "d257c394-b3e4-4104-9a4c-1a18decd7c1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:26:24.974760 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:24.974737    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d257c394-b3e4-4104-9a4c-1a18decd7c1f-kube-api-access-7lpmg" (OuterVolumeSpecName: "kube-api-access-7lpmg") pod "d257c394-b3e4-4104-9a4c-1a18decd7c1f" (UID: "d257c394-b3e4-4104-9a4c-1a18decd7c1f"). InnerVolumeSpecName "kube-api-access-7lpmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:26:25.073388 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.073337    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb0718a-73c9-4135-860f-c7fb09628239-plugins-config-volume\") pod \"9fb0718a-73c9-4135-860f-c7fb09628239\" (UID: \"9fb0718a-73c9-4135-860f-c7fb09628239\") "
Aug 04 21:26:25.073560 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.073469    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwds9\" (UniqueName: \"kubernetes.io/projected/9fb0718a-73c9-4135-860f-c7fb09628239-kube-api-access-kwds9\") pod \"9fb0718a-73c9-4135-860f-c7fb09628239\" (UID: \"9fb0718a-73c9-4135-860f-c7fb09628239\") "
Aug 04 21:26:25.073718 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.073702    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lpmg\" (UniqueName: \"kubernetes.io/projected/d257c394-b3e4-4104-9a4c-1a18decd7c1f-kube-api-access-7lpmg\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:26:25.073780 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.073724    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d257c394-b3e4-4104-9a4c-1a18decd7c1f-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:26:25.073780 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.073700    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb0718a-73c9-4135-860f-c7fb09628239-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9fb0718a-73c9-4135-860f-c7fb09628239" (UID: "9fb0718a-73c9-4135-860f-c7fb09628239"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:26:25.075459 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.075436    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb0718a-73c9-4135-860f-c7fb09628239-kube-api-access-kwds9" (OuterVolumeSpecName: "kube-api-access-kwds9") pod "9fb0718a-73c9-4135-860f-c7fb09628239" (UID: "9fb0718a-73c9-4135-860f-c7fb09628239"). InnerVolumeSpecName "kube-api-access-kwds9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:26:25.174591 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.174526    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwds9\" (UniqueName: \"kubernetes.io/projected/9fb0718a-73c9-4135-860f-c7fb09628239-kube-api-access-kwds9\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:26:25.174591 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.174548    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb0718a-73c9-4135-860f-c7fb09628239-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:26:25.760969 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.760936    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="295b49cb-dfa2-4c01-bc2f-b10605bcdf9c" path="/var/lib/kubelet/pods/295b49cb-dfa2-4c01-bc2f-b10605bcdf9c/volumes"
Aug 04 21:26:25.849309 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.849283    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"
Aug 04 21:26:25.849478 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.849315    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf" event={"ID":"d257c394-b3e4-4104-9a4c-1a18decd7c1f","Type":"ContainerDied","Data":"23acc4253653c0a3f499234510b2d58f52d22114c22b96f5c9d181e139ab314c"}
Aug 04 21:26:25.849478 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.849398    2565 scope.go:117] "RemoveContainer" containerID="b188ae9788667164ecc23f5634ecde288ba3413bc3530e5fed0634a7786d9c00"
Aug 04 21:26:25.850754 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.850735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz" event={"ID":"9fb0718a-73c9-4135-860f-c7fb09628239","Type":"ContainerDied","Data":"94d425c7a4f7db1cf85cc8dfe6ed88ea8bcbd0ad96c9e999382ff9293bc7cb75"}
Aug 04 21:26:25.850863 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.850769    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"
Aug 04 21:26:25.858317 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.858287    2565 scope.go:117] "RemoveContainer" containerID="7ccfb77a65c02f55b5628c1521557252de5fc7068f14e7752528b60cce923537"
Aug 04 21:26:25.867072 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.867052    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"]
Aug 04 21:26:25.870907 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.870882    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4ea15-7bdb4675dd-7v2rz"]
Aug 04 21:26:25.883705 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.883682    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"]
Aug 04 21:26:25.885314 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:25.885292    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4ea15-667f47c9b-fv7bf"]
Aug 04 21:26:26.561469 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:26.561441    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:26:27.760134 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:27.760100    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fb0718a-73c9-4135-860f-c7fb09628239" path="/var/lib/kubelet/pods/9fb0718a-73c9-4135-860f-c7fb09628239/volumes"
Aug 04 21:26:27.760513 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:27.760501    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d257c394-b3e4-4104-9a4c-1a18decd7c1f" path="/var/lib/kubelet/pods/d257c394-b3e4-4104-9a4c-1a18decd7c1f/volumes"
Aug 04 21:26:37.350133 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.350098    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"]
Aug 04 21:26:37.350661 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.350514    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb0718a-73c9-4135-860f-c7fb09628239" containerName="payload-pre-processing"
Aug 04 21:26:37.350661 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.350527    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb0718a-73c9-4135-860f-c7fb09628239" containerName="payload-pre-processing"
Aug 04 21:26:37.350661 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.350536    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d257c394-b3e4-4104-9a4c-1a18decd7c1f" containerName="payload-processing"
Aug 04 21:26:37.350661 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.350542    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d257c394-b3e4-4104-9a4c-1a18decd7c1f" containerName="payload-processing"
Aug 04 21:26:37.350661 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.350551    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295b49cb-dfa2-4c01-bc2f-b10605bcdf9c" containerName="revoke-keys"
Aug 04 21:26:37.350661 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.350559    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="295b49cb-dfa2-4c01-bc2f-b10605bcdf9c" containerName="revoke-keys"
Aug 04 21:26:37.350661 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.350648    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb0718a-73c9-4135-860f-c7fb09628239" containerName="payload-pre-processing"
Aug 04 21:26:37.350661 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.350660    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="295b49cb-dfa2-4c01-bc2f-b10605bcdf9c" containerName="revoke-keys"
Aug 04 21:26:37.350963 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.350669    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d257c394-b3e4-4104-9a4c-1a18decd7c1f" containerName="payload-processing"
Aug 04 21:26:37.353288 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.353273    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:26:37.355668 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.355645    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-a189ec\""
Aug 04 21:26:37.355794 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.355711    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-a189ec-dockercfg-5gfgl\""
Aug 04 21:26:37.363847 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.363820    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"]
Aug 04 21:26:37.396385 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.396331    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"]
Aug 04 21:26:37.400973 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.400943    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"]
Aug 04 21:26:37.401141 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.401069    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:26:37.480821 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.480783    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc445817-01fa-4642-a93e-2fd34f95fcae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x\" (UID: \"dc445817-01fa-4642-a93e-2fd34f95fcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:26:37.480976 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.480838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd2jt\" (UniqueName: \"kubernetes.io/projected/dc445817-01fa-4642-a93e-2fd34f95fcae-kube-api-access-cd2jt\") pod \"payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x\" (UID: \"dc445817-01fa-4642-a93e-2fd34f95fcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:26:37.480976 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.480959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwdn\" (UniqueName: \"kubernetes.io/projected/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-kube-api-access-wqwdn\") pod \"payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx\" (UID: \"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:26:37.481057 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.481018    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx\" (UID: \"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:26:37.581802 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.581763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd2jt\" (UniqueName: \"kubernetes.io/projected/dc445817-01fa-4642-a93e-2fd34f95fcae-kube-api-access-cd2jt\") pod \"payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x\" (UID: \"dc445817-01fa-4642-a93e-2fd34f95fcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:26:37.581973 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.581856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwdn\" (UniqueName: \"kubernetes.io/projected/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-kube-api-access-wqwdn\") pod \"payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx\" (UID: \"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:26:37.581973 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.581911    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx\" (UID: \"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:26:37.581973 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.581962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc445817-01fa-4642-a93e-2fd34f95fcae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x\" (UID: \"dc445817-01fa-4642-a93e-2fd34f95fcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:26:37.582664 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.582644    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx\" (UID: \"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:26:37.582712 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.582651    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc445817-01fa-4642-a93e-2fd34f95fcae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x\" (UID: \"dc445817-01fa-4642-a93e-2fd34f95fcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:26:37.589880 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.589853    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd2jt\" (UniqueName: \"kubernetes.io/projected/dc445817-01fa-4642-a93e-2fd34f95fcae-kube-api-access-cd2jt\") pod \"payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x\" (UID: \"dc445817-01fa-4642-a93e-2fd34f95fcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:26:37.590001 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.589864    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwdn\" (UniqueName: \"kubernetes.io/projected/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-kube-api-access-wqwdn\") pod \"payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx\" (UID: \"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:26:37.664662 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.664575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:26:37.714370 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.713956    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:26:37.811488 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.811462    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"]
Aug 04 21:26:37.813523 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:26:37.813497    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc445817_01fa_4642_a93e_2fd34f95fcae.slice/crio-524084d0d503331366d803b95657114077af3d27f633769ca84cd06ea377b32d WatchSource:0}: Error finding container 524084d0d503331366d803b95657114077af3d27f633769ca84cd06ea377b32d: Status 404 returned error can't find the container with id 524084d0d503331366d803b95657114077af3d27f633769ca84cd06ea377b32d
Aug 04 21:26:37.872844 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.872709    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"]
Aug 04 21:26:37.875153 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:26:37.875125    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4c0f10_193b_4fbb_a4d2_b51a27ec82f4.slice/crio-d21380b3a53a1240c53ee09210da2d38c54de426a9646da1e63a52af47f3adb6 WatchSource:0}: Error finding container d21380b3a53a1240c53ee09210da2d38c54de426a9646da1e63a52af47f3adb6: Status 404 returned error can't find the container with id d21380b3a53a1240c53ee09210da2d38c54de426a9646da1e63a52af47f3adb6
Aug 04 21:26:37.906594 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.906568    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx" event={"ID":"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4","Type":"ContainerStarted","Data":"d21380b3a53a1240c53ee09210da2d38c54de426a9646da1e63a52af47f3adb6"}
Aug 04 21:26:37.907883 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.907861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x" event={"ID":"dc445817-01fa-4642-a93e-2fd34f95fcae","Type":"ContainerStarted","Data":"91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735"}
Aug 04 21:26:37.907995 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.907899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x" event={"ID":"dc445817-01fa-4642-a93e-2fd34f95fcae","Type":"ContainerStarted","Data":"524084d0d503331366d803b95657114077af3d27f633769ca84cd06ea377b32d"}
Aug 04 21:26:37.907995 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.907962    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:26:37.926069 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:37.926029    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x" podStartSLOduration=0.92601508 podStartE2EDuration="926.01508ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:26:37.923781305 +0000 UTC m=+2772.675887284" watchObservedRunningTime="2026-08-04 21:26:37.92601508 +0000 UTC m=+2772.678121056"
Aug 04 21:26:38.914893 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:38.914860    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx" event={"ID":"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4","Type":"ContainerStarted","Data":"5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3"}
Aug 04 21:26:38.915387 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:38.914998    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:26:38.932706 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:38.932654    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx" podStartSLOduration=1.9326388570000002 podStartE2EDuration="1.932638857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:26:38.930678659 +0000 UTC m=+2773.682784659" watchObservedRunningTime="2026-08-04 21:26:38.932638857 +0000 UTC m=+2773.684744834"
Aug 04 21:26:39.151123 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:39.151086    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:26:48.917834 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:48.917759    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:26:49.922190 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:26:49.922157    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:27:05.693248 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.693215    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"]
Aug 04 21:27:05.696784 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.696768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:27:05.699135 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.699110    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-cd644a\""
Aug 04 21:27:05.699287 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.699164    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-cd644a-dockercfg-ccwb7\""
Aug 04 21:27:05.706699 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.706650    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"]
Aug 04 21:27:05.717084 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.717060    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"]
Aug 04 21:27:05.721205 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.721185    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:27:05.727180 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.727157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/548e47c8-8155-4da6-9d48-a0e340ea2ff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428\" (UID: \"548e47c8-8155-4da6-9d48-a0e340ea2ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:27:05.727297 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.727201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frc4g\" (UniqueName: \"kubernetes.io/projected/548e47c8-8155-4da6-9d48-a0e340ea2ff9-kube-api-access-frc4g\") pod \"payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428\" (UID: \"548e47c8-8155-4da6-9d48-a0e340ea2ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:27:05.731150 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.731129    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"]
Aug 04 21:27:05.828495 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.828468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5278aaa2-65c9-497e-bd05-5388aa4f167c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5\" (UID: \"5278aaa2-65c9-497e-bd05-5388aa4f167c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:27:05.828632 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.828540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/548e47c8-8155-4da6-9d48-a0e340ea2ff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428\" (UID: \"548e47c8-8155-4da6-9d48-a0e340ea2ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:27:05.828632 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.828568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frc4g\" (UniqueName: \"kubernetes.io/projected/548e47c8-8155-4da6-9d48-a0e340ea2ff9-kube-api-access-frc4g\") pod \"payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428\" (UID: \"548e47c8-8155-4da6-9d48-a0e340ea2ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:27:05.828715 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.828650    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvkt\" (UniqueName: \"kubernetes.io/projected/5278aaa2-65c9-497e-bd05-5388aa4f167c-kube-api-access-rvvkt\") pod \"payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5\" (UID: \"5278aaa2-65c9-497e-bd05-5388aa4f167c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:27:05.829295 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.829276    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/548e47c8-8155-4da6-9d48-a0e340ea2ff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428\" (UID: \"548e47c8-8155-4da6-9d48-a0e340ea2ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:27:05.837289 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.837259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frc4g\" (UniqueName: \"kubernetes.io/projected/548e47c8-8155-4da6-9d48-a0e340ea2ff9-kube-api-access-frc4g\") pod \"payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428\" (UID: \"548e47c8-8155-4da6-9d48-a0e340ea2ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:27:05.930013 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.929990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvkt\" (UniqueName: \"kubernetes.io/projected/5278aaa2-65c9-497e-bd05-5388aa4f167c-kube-api-access-rvvkt\") pod \"payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5\" (UID: \"5278aaa2-65c9-497e-bd05-5388aa4f167c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:27:05.930120 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.930035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5278aaa2-65c9-497e-bd05-5388aa4f167c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5\" (UID: \"5278aaa2-65c9-497e-bd05-5388aa4f167c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:27:05.930632 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.930615    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5278aaa2-65c9-497e-bd05-5388aa4f167c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5\" (UID: \"5278aaa2-65c9-497e-bd05-5388aa4f167c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:27:05.939020 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:05.939000    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvkt\" (UniqueName: \"kubernetes.io/projected/5278aaa2-65c9-497e-bd05-5388aa4f167c-kube-api-access-rvvkt\") pod \"payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5\" (UID: \"5278aaa2-65c9-497e-bd05-5388aa4f167c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:27:06.007967 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:06.007947    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:27:06.031616 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:06.031594    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:27:06.153480 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:06.153419    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"]
Aug 04 21:27:06.155710 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:27:06.155682    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548e47c8_8155_4da6_9d48_a0e340ea2ff9.slice/crio-44851e44dacaa7cb7cd7ee3d1802ac7d0d61951f8cab36956864dcf904ab2d50 WatchSource:0}: Error finding container 44851e44dacaa7cb7cd7ee3d1802ac7d0d61951f8cab36956864dcf904ab2d50: Status 404 returned error can't find the container with id 44851e44dacaa7cb7cd7ee3d1802ac7d0d61951f8cab36956864dcf904ab2d50
Aug 04 21:27:06.177262 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:06.177240    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"]
Aug 04 21:27:06.178219 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:27:06.178197    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5278aaa2_65c9_497e_bd05_5388aa4f167c.slice/crio-bb61ae90f361243546fa18bc64500480394a8b3afb86b09d0ed3d8793350017c WatchSource:0}: Error finding container bb61ae90f361243546fa18bc64500480394a8b3afb86b09d0ed3d8793350017c: Status 404 returned error can't find the container with id bb61ae90f361243546fa18bc64500480394a8b3afb86b09d0ed3d8793350017c
Aug 04 21:27:07.059892 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:07.059857    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5" event={"ID":"5278aaa2-65c9-497e-bd05-5388aa4f167c","Type":"ContainerStarted","Data":"09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc"}
Aug 04 21:27:07.060362 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:07.059918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5" event={"ID":"5278aaa2-65c9-497e-bd05-5388aa4f167c","Type":"ContainerStarted","Data":"bb61ae90f361243546fa18bc64500480394a8b3afb86b09d0ed3d8793350017c"}
Aug 04 21:27:07.060362 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:07.060037    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:27:07.061366 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:07.061319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428" event={"ID":"548e47c8-8155-4da6-9d48-a0e340ea2ff9","Type":"ContainerStarted","Data":"6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d"}
Aug 04 21:27:07.061499 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:07.061374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428" event={"ID":"548e47c8-8155-4da6-9d48-a0e340ea2ff9","Type":"ContainerStarted","Data":"44851e44dacaa7cb7cd7ee3d1802ac7d0d61951f8cab36956864dcf904ab2d50"}
Aug 04 21:27:07.061499 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:07.061406    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:27:07.081577 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:07.081535    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5" podStartSLOduration=2.081522535 podStartE2EDuration="2.081522535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:27:07.079944042 +0000 UTC m=+2801.832050020" watchObservedRunningTime="2026-08-04 21:27:07.081522535 +0000 UTC m=+2801.833628512"
Aug 04 21:27:07.096567 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:07.096527    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428" podStartSLOduration=2.096515685 podStartE2EDuration="2.096515685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:27:07.094548351 +0000 UTC m=+2801.846654329" watchObservedRunningTime="2026-08-04 21:27:07.096515685 +0000 UTC m=+2801.848621678"
Aug 04 21:27:07.946355 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:07.946309    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:27:18.069523 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:18.069495    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:27:18.070035 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:18.069560    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:27:39.250293 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.250255    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"]
Aug 04 21:27:39.254510 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.254495    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.257322 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.257260    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a189ec\"/\"kube-root-ca.crt\""
Aug 04 21:27:39.257825 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.257806    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a189ec\"/\"openshift-service-ca.crt\""
Aug 04 21:27:39.257930 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.257876    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a189ec\"/\"auth-test-model-a189ec-kserve-self-signed-certs\""
Aug 04 21:27:39.257930 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.257810    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a189ec\"/\"default-dockercfg-8rzqr\""
Aug 04 21:27:39.263373 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.263334    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"]
Aug 04 21:27:39.335694 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.335662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372756e5-9f90-4b26-bc06-c9278717efcb-tls-certs\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.335850 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.335707    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-home\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.335850 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.335760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-dshm\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.335850 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.335830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-model-cache\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.335981 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.335851    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-tmp-dir\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.335981 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.335872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2n9\" (UniqueName: \"kubernetes.io/projected/372756e5-9f90-4b26-bc06-c9278717efcb-kube-api-access-rf2n9\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.436417 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.436383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-dshm\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.436580 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.436485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-model-cache\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.436580 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.436510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-tmp-dir\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.436580 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.436526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2n9\" (UniqueName: \"kubernetes.io/projected/372756e5-9f90-4b26-bc06-c9278717efcb-kube-api-access-rf2n9\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.436722 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.436600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372756e5-9f90-4b26-bc06-c9278717efcb-tls-certs\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.436722 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.436651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-home\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.436929 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.436898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-model-cache\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.437009 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.436923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-tmp-dir\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.437009 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.436978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-home\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.438542 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.438514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-dshm\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.438886 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.438869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372756e5-9f90-4b26-bc06-c9278717efcb-tls-certs\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.444576 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.444554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2n9\" (UniqueName: \"kubernetes.io/projected/372756e5-9f90-4b26-bc06-c9278717efcb-kube-api-access-rf2n9\") pod \"auth-test-model-a189ec-kserve-555968c4c5-dnqrz\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") " pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.564212 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.564120    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:39.701556 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:39.701528    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"]
Aug 04 21:27:39.703695 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:27:39.703668    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod372756e5_9f90_4b26_bc06_c9278717efcb.slice/crio-0b5df86b2d07120716a5d92b3654502b67152967e02f5c7c7911a64e4972f4d2 WatchSource:0}: Error finding container 0b5df86b2d07120716a5d92b3654502b67152967e02f5c7c7911a64e4972f4d2: Status 404 returned error can't find the container with id 0b5df86b2d07120716a5d92b3654502b67152967e02f5c7c7911a64e4972f4d2
Aug 04 21:27:40.245718 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:40.245683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz" event={"ID":"372756e5-9f90-4b26-bc06-c9278717efcb","Type":"ContainerStarted","Data":"7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4"}
Aug 04 21:27:40.245718 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:40.245724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz" event={"ID":"372756e5-9f90-4b26-bc06-c9278717efcb","Type":"ContainerStarted","Data":"0b5df86b2d07120716a5d92b3654502b67152967e02f5c7c7911a64e4972f4d2"}
Aug 04 21:27:40.264407 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:40.264358    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz" podStartSLOduration=1.264326743 podStartE2EDuration="1.264326743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:27:40.263195468 +0000 UTC m=+2835.015301446" watchObservedRunningTime="2026-08-04 21:27:40.264326743 +0000 UTC m=+2835.016432720"
Aug 04 21:27:41.745639 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:41.745604    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:27:49.565230 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:49.565192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:49.565744 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:49.565245    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:49.577683 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:49.577656    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:27:50.307860 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:27:50.307836    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:28:02.660135 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.660102    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"]
Aug 04 21:28:02.664062 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.664043    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.666634 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.666616    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd644a\"/\"openshift-service-ca.crt\""
Aug 04 21:28:02.666731 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.666630    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd644a\"/\"auth-test-model-cd644a-kserve-self-signed-certs\""
Aug 04 21:28:02.667648 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.667633    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd644a\"/\"default-dockercfg-ngjrk\""
Aug 04 21:28:02.667706 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.667690    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd644a\"/\"kube-root-ca.crt\""
Aug 04 21:28:02.674176 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.674155    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"]
Aug 04 21:28:02.744381 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.744335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxv8g\" (UniqueName: \"kubernetes.io/projected/3b27682c-a4cf-4b4b-9121-937e766d5d20-kube-api-access-nxv8g\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.744542 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.744419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b27682c-a4cf-4b4b-9121-937e766d5d20-tls-certs\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.744542 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.744443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-tmp-dir\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.744542 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.744525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-home\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.744651 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.744597    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-model-cache\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.744651 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.744622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-dshm\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.846006 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.845980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-tmp-dir\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.846197 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.846036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-home\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.846197 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.846105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-model-cache\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.846197 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.846132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-dshm\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.846197 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.846157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxv8g\" (UniqueName: \"kubernetes.io/projected/3b27682c-a4cf-4b4b-9121-937e766d5d20-kube-api-access-nxv8g\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.846467 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.846233    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b27682c-a4cf-4b4b-9121-937e766d5d20-tls-certs\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.846467 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.846416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-tmp-dir\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.846584 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.846459    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-home\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.846584 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.846555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-model-cache\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.848363 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.848330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-dshm\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.848640 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.848623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b27682c-a4cf-4b4b-9121-937e766d5d20-tls-certs\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.856088 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.856065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxv8g\" (UniqueName: \"kubernetes.io/projected/3b27682c-a4cf-4b4b-9121-937e766d5d20-kube-api-access-nxv8g\") pod \"auth-test-model-cd644a-kserve-77cd898f79-j78w5\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") " pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:02.974473 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:02.974439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:03.114571 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:03.114544    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"]
Aug 04 21:28:03.117540 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:28:03.117510    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b27682c_a4cf_4b4b_9121_937e766d5d20.slice/crio-43fc13249966079ee61732ccd355af18776316ff14475f19c03af9bbc0b1dac6 WatchSource:0}: Error finding container 43fc13249966079ee61732ccd355af18776316ff14475f19c03af9bbc0b1dac6: Status 404 returned error can't find the container with id 43fc13249966079ee61732ccd355af18776316ff14475f19c03af9bbc0b1dac6
Aug 04 21:28:03.369130 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:03.369030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5" event={"ID":"3b27682c-a4cf-4b4b-9121-937e766d5d20","Type":"ContainerStarted","Data":"8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a"}
Aug 04 21:28:03.369130 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:03.369073    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5" event={"ID":"3b27682c-a4cf-4b4b-9121-937e766d5d20","Type":"ContainerStarted","Data":"43fc13249966079ee61732ccd355af18776316ff14475f19c03af9bbc0b1dac6"}
Aug 04 21:28:03.391565 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:03.391515    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5" podStartSLOduration=1.391500911 podStartE2EDuration="1.391500911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:28:03.390130413 +0000 UTC m=+2858.142236391" watchObservedRunningTime="2026-08-04 21:28:03.391500911 +0000 UTC m=+2858.143606888"
Aug 04 21:28:04.966169 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:04.966136    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:28:12.975464 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:12.975429    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:12.975987 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:12.975489    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:12.988119 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:12.988093    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:13.438202 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:13.438117    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:28:28.960555 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:28.960523    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:28:42.650227 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:42.650194    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:28:53.770841 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:28:53.770805    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:29:09.265820 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:29:09.265785    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:29:20.943822 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:29:20.943784    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:29:35.550332 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:29:35.550293    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:29:47.258411 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:29:47.258374    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:30:02.252798 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:30:02.252744    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:30:13.957446 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:30:13.957409    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:30:25.959414 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:30:25.959387    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:30:25.965221 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:30:25.965200    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:30:28.951631 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:30:28.951600    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:30:40.660591 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:30:40.660552    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:30:55.748969 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:30:55.748932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:30:59.413809 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:30:59.413776    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"]
Aug 04 21:31:00.271550 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:00.271523    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"]
Aug 04 21:31:05.245097 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:05.245067    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:31:13.259812 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:13.259779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:31:14.433020 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.432964    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz" podUID="372756e5-9f90-4b26-bc06-c9278717efcb" containerName="main" containerID="cri-o://7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4" gracePeriod=45
Aug 04 21:31:14.683591 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.683533    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:31:14.778030 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778001    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-home\") pod \"372756e5-9f90-4b26-bc06-c9278717efcb\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") "
Aug 04 21:31:14.778193 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778090    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf2n9\" (UniqueName: \"kubernetes.io/projected/372756e5-9f90-4b26-bc06-c9278717efcb-kube-api-access-rf2n9\") pod \"372756e5-9f90-4b26-bc06-c9278717efcb\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") "
Aug 04 21:31:14.778193 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778115    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372756e5-9f90-4b26-bc06-c9278717efcb-tls-certs\") pod \"372756e5-9f90-4b26-bc06-c9278717efcb\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") "
Aug 04 21:31:14.778193 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778139    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-tmp-dir\") pod \"372756e5-9f90-4b26-bc06-c9278717efcb\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") "
Aug 04 21:31:14.778193 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778162    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-dshm\") pod \"372756e5-9f90-4b26-bc06-c9278717efcb\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") "
Aug 04 21:31:14.778446 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778300    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-home" (OuterVolumeSpecName: "home") pod "372756e5-9f90-4b26-bc06-c9278717efcb" (UID: "372756e5-9f90-4b26-bc06-c9278717efcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:31:14.778446 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778312    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-model-cache\") pod \"372756e5-9f90-4b26-bc06-c9278717efcb\" (UID: \"372756e5-9f90-4b26-bc06-c9278717efcb\") "
Aug 04 21:31:14.778561 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778470    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "372756e5-9f90-4b26-bc06-c9278717efcb" (UID: "372756e5-9f90-4b26-bc06-c9278717efcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:31:14.778561 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778521    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "372756e5-9f90-4b26-bc06-c9278717efcb" (UID: "372756e5-9f90-4b26-bc06-c9278717efcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:31:14.778764 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778742    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:14.778764 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778761    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:14.778764 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.778769    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:14.780387 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.780365    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-dshm" (OuterVolumeSpecName: "dshm") pod "372756e5-9f90-4b26-bc06-c9278717efcb" (UID: "372756e5-9f90-4b26-bc06-c9278717efcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:31:14.780483 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.780469    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372756e5-9f90-4b26-bc06-c9278717efcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "372756e5-9f90-4b26-bc06-c9278717efcb" (UID: "372756e5-9f90-4b26-bc06-c9278717efcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 21:31:14.780525 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.780488    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372756e5-9f90-4b26-bc06-c9278717efcb-kube-api-access-rf2n9" (OuterVolumeSpecName: "kube-api-access-rf2n9") pod "372756e5-9f90-4b26-bc06-c9278717efcb" (UID: "372756e5-9f90-4b26-bc06-c9278717efcb"). InnerVolumeSpecName "kube-api-access-rf2n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:31:14.879546 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.879524    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf2n9\" (UniqueName: \"kubernetes.io/projected/372756e5-9f90-4b26-bc06-c9278717efcb-kube-api-access-rf2n9\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:14.879546 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.879546    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372756e5-9f90-4b26-bc06-c9278717efcb-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:14.879700 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:14.879556    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372756e5-9f90-4b26-bc06-c9278717efcb-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:15.290370 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.290322    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5" podUID="3b27682c-a4cf-4b4b-9121-937e766d5d20" containerName="main" containerID="cri-o://8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a" gracePeriod=45
Aug 04 21:31:15.346746 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.346718    2565 generic.go:356] "Generic (PLEG): container finished" podID="372756e5-9f90-4b26-bc06-c9278717efcb" containerID="7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4" exitCode=0
Aug 04 21:31:15.346862 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.346755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz" event={"ID":"372756e5-9f90-4b26-bc06-c9278717efcb","Type":"ContainerDied","Data":"7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4"}
Aug 04 21:31:15.346862 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.346768    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"
Aug 04 21:31:15.346862 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.346784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz" event={"ID":"372756e5-9f90-4b26-bc06-c9278717efcb","Type":"ContainerDied","Data":"0b5df86b2d07120716a5d92b3654502b67152967e02f5c7c7911a64e4972f4d2"}
Aug 04 21:31:15.346862 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.346805    2565 scope.go:117] "RemoveContainer" containerID="7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4"
Aug 04 21:31:15.395957 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.395928    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"]
Aug 04 21:31:15.405411 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.401625    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/auth-test-model-a189ec-kserve-555968c4c5-dnqrz"]
Aug 04 21:31:15.405411 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.404989    2565 scope.go:117] "RemoveContainer" containerID="7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4"
Aug 04 21:31:15.405719 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:31:15.405663    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4\": container with ID starting with 7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4 not found: ID does not exist" containerID="7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4"
Aug 04 21:31:15.405770 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.405706    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4"} err="failed to get container status \"7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4\": rpc error: code = NotFound desc = could not find container \"7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4\": container with ID starting with 7737dce9a0db2dd6b3346a3c54e93dc87ba791d756acfd426a15e5387c1549c4 not found: ID does not exist"
Aug 04 21:31:15.555039 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.554980    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:31:15.687792 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.687765    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-tmp-dir\") pod \"3b27682c-a4cf-4b4b-9121-937e766d5d20\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") "
Aug 04 21:31:15.687953 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.687803    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-dshm\") pod \"3b27682c-a4cf-4b4b-9121-937e766d5d20\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") "
Aug 04 21:31:15.687953 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.687849    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxv8g\" (UniqueName: \"kubernetes.io/projected/3b27682c-a4cf-4b4b-9121-937e766d5d20-kube-api-access-nxv8g\") pod \"3b27682c-a4cf-4b4b-9121-937e766d5d20\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") "
Aug 04 21:31:15.687953 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.687888    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-home\") pod \"3b27682c-a4cf-4b4b-9121-937e766d5d20\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") "
Aug 04 21:31:15.687953 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.687916    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-model-cache\") pod \"3b27682c-a4cf-4b4b-9121-937e766d5d20\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") "
Aug 04 21:31:15.688286 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.687991    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b27682c-a4cf-4b4b-9121-937e766d5d20-tls-certs\") pod \"3b27682c-a4cf-4b4b-9121-937e766d5d20\" (UID: \"3b27682c-a4cf-4b4b-9121-937e766d5d20\") "
Aug 04 21:31:15.688286 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.687995    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b27682c-a4cf-4b4b-9121-937e766d5d20" (UID: "3b27682c-a4cf-4b4b-9121-937e766d5d20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:31:15.688286 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.688182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-home" (OuterVolumeSpecName: "home") pod "3b27682c-a4cf-4b4b-9121-937e766d5d20" (UID: "3b27682c-a4cf-4b4b-9121-937e766d5d20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:31:15.688286 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.688216    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b27682c-a4cf-4b4b-9121-937e766d5d20" (UID: "3b27682c-a4cf-4b4b-9121-937e766d5d20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:31:15.688506 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.688301    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:15.688506 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.688320    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:15.688506 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.688388    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:15.690126 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.690099    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-dshm" (OuterVolumeSpecName: "dshm") pod "3b27682c-a4cf-4b4b-9121-937e766d5d20" (UID: "3b27682c-a4cf-4b4b-9121-937e766d5d20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:31:15.690218 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.690122    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b27682c-a4cf-4b4b-9121-937e766d5d20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b27682c-a4cf-4b4b-9121-937e766d5d20" (UID: "3b27682c-a4cf-4b4b-9121-937e766d5d20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 21:31:15.690218 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.690166    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b27682c-a4cf-4b4b-9121-937e766d5d20-kube-api-access-nxv8g" (OuterVolumeSpecName: "kube-api-access-nxv8g") pod "3b27682c-a4cf-4b4b-9121-937e766d5d20" (UID: "3b27682c-a4cf-4b4b-9121-937e766d5d20"). InnerVolumeSpecName "kube-api-access-nxv8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:31:15.760936 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.760909    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="372756e5-9f90-4b26-bc06-c9278717efcb" path="/var/lib/kubelet/pods/372756e5-9f90-4b26-bc06-c9278717efcb/volumes"
Aug 04 21:31:15.788913 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.788887    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b27682c-a4cf-4b4b-9121-937e766d5d20-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:15.788913 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.788911    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxv8g\" (UniqueName: \"kubernetes.io/projected/3b27682c-a4cf-4b4b-9121-937e766d5d20-kube-api-access-nxv8g\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:15.789060 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:15.788923    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b27682c-a4cf-4b4b-9121-937e766d5d20-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:31:16.352594 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:16.352555    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b27682c-a4cf-4b4b-9121-937e766d5d20" containerID="8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a" exitCode=0
Aug 04 21:31:16.352802 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:16.352600    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"
Aug 04 21:31:16.352802 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:16.352645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5" event={"ID":"3b27682c-a4cf-4b4b-9121-937e766d5d20","Type":"ContainerDied","Data":"8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a"}
Aug 04 21:31:16.352802 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:16.352696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5" event={"ID":"3b27682c-a4cf-4b4b-9121-937e766d5d20","Type":"ContainerDied","Data":"43fc13249966079ee61732ccd355af18776316ff14475f19c03af9bbc0b1dac6"}
Aug 04 21:31:16.352802 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:16.352721    2565 scope.go:117] "RemoveContainer" containerID="8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a"
Aug 04 21:31:16.361144 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:16.361119    2565 scope.go:117] "RemoveContainer" containerID="8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a"
Aug 04 21:31:16.361403 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:31:16.361382    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a\": container with ID starting with 8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a not found: ID does not exist" containerID="8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a"
Aug 04 21:31:16.361475 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:16.361410    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a"} err="failed to get container status \"8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a\": rpc error: code = NotFound desc = could not find container \"8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a\": container with ID starting with 8e3ed5a3642969c1d7d299c45a1a424d47640249c61e4f6a5fc6b2d07e3e7a8a not found: ID does not exist"
Aug 04 21:31:16.376712 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:16.376688    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"]
Aug 04 21:31:16.386134 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:16.386115    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/auth-test-model-cd644a-kserve-77cd898f79-j78w5"]
Aug 04 21:31:17.761846 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:17.761814    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b27682c-a4cf-4b4b-9121-937e766d5d20" path="/var/lib/kubelet/pods/3b27682c-a4cf-4b4b-9121-937e766d5d20/volumes"
Aug 04 21:31:26.139844 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:26.139811    2565 scope.go:117] "RemoveContainer" containerID="18c3ae8d74162515945fb5250a69aa5758db2d6f307441999fda35e34374d699"
Aug 04 21:31:42.262529 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.262497    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"]
Aug 04 21:31:42.262960 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.262878    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372756e5-9f90-4b26-bc06-c9278717efcb" containerName="main"
Aug 04 21:31:42.262960 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.262888    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="372756e5-9f90-4b26-bc06-c9278717efcb" containerName="main"
Aug 04 21:31:42.262960 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.262913    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b27682c-a4cf-4b4b-9121-937e766d5d20" containerName="main"
Aug 04 21:31:42.262960 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.262919    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b27682c-a4cf-4b4b-9121-937e766d5d20" containerName="main"
Aug 04 21:31:42.263091 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.262976    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b27682c-a4cf-4b4b-9121-937e766d5d20" containerName="main"
Aug 04 21:31:42.263091 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.262985    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="372756e5-9f90-4b26-bc06-c9278717efcb" containerName="main"
Aug 04 21:31:42.265824 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.265791    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.269198 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.269177    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd644a\"/\"kube-root-ca.crt\""
Aug 04 21:31:42.271420 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.271399    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd644a\"/\"openshift-service-ca.crt\""
Aug 04 21:31:42.271874 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.271674    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd644a\"/\"sub-test-model-cd644a-kserve-self-signed-certs\""
Aug 04 21:31:42.271874 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.271775    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd644a\"/\"default-dockercfg-ngjrk\""
Aug 04 21:31:42.274648 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.274625    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"]
Aug 04 21:31:42.324594 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.324567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tls-certs\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.324708 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.324610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-dshm\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.324708 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.324674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-model-cache\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.324787 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.324729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tmp-dir\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.324787 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.324757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2lcz\" (UniqueName: \"kubernetes.io/projected/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-kube-api-access-k2lcz\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.324881 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.324787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-home\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.425970 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.425928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tls-certs\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.425970 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.425977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-dshm\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.426203 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.426025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-model-cache\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.426203 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.426061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tmp-dir\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.426203 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.426097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2lcz\" (UniqueName: \"kubernetes.io/projected/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-kube-api-access-k2lcz\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.426203 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.426138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-home\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.426517 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.426494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-home\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.426611 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.426564    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tmp-dir\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.426611 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.426587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-model-cache\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.428397 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.428376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-dshm\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.428579 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.428564    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tls-certs\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.434355 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.434322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2lcz\" (UniqueName: \"kubernetes.io/projected/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-kube-api-access-k2lcz\") pod \"sub-test-model-cd644a-kserve-846d6977f5-nmx8q\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") " pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.575977 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.575885    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:42.713533 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:42.713510    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"]
Aug 04 21:31:42.715549 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:31:42.715523    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc426fd5_4e35_4a72_93ae_76c1ebeafaaa.slice/crio-8d1c36fd1fdabc026ad0da232b500d618cc3ca58d1c05ec60791a577bee76495 WatchSource:0}: Error finding container 8d1c36fd1fdabc026ad0da232b500d618cc3ca58d1c05ec60791a577bee76495: Status 404 returned error can't find the container with id 8d1c36fd1fdabc026ad0da232b500d618cc3ca58d1c05ec60791a577bee76495
Aug 04 21:31:43.490031 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:43.489999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q" event={"ID":"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa","Type":"ContainerStarted","Data":"08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8"}
Aug 04 21:31:43.490446 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:43.490040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q" event={"ID":"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa","Type":"ContainerStarted","Data":"8d1c36fd1fdabc026ad0da232b500d618cc3ca58d1c05ec60791a577bee76495"}
Aug 04 21:31:43.507433 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:43.507386    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q" podStartSLOduration=1.507370452 podStartE2EDuration="1.507370452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:31:43.506892284 +0000 UTC m=+3078.258998270" watchObservedRunningTime="2026-08-04 21:31:43.507370452 +0000 UTC m=+3078.259476420"
Aug 04 21:31:44.752101 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:44.752063    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:31:52.576615 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:52.576581    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:52.576986 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:52.576629    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:52.589858 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:52.589834    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:31:53.554947 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:31:53.554917    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:32:26.170534 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:32:26.170496    2565 scope.go:117] "RemoveContainer" containerID="309c59e77bc5751f311a9edb1d547b4517a19075889043453fe39c8e0c134964"
Aug 04 21:32:26.178718 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:32:26.178696    2565 scope.go:117] "RemoveContainer" containerID="5902bd6422c854c388a929a7959c221ae4200165eafcf800a474a67134581545"
Aug 04 21:32:28.261216 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:32:28.261183    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:32:40.667558 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:32:40.667527    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:32:51.457875 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:32:51.457797    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:32:57.257563 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:32:57.257533    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"]
Aug 04 21:33:01.858113 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:01.858070    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:33:10.167591 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.167557    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"]
Aug 04 21:33:10.171224 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.171202    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.173810 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.173782    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a189ec\"/\"kube-root-ca.crt\""
Aug 04 21:33:10.173810 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.173784    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a189ec\"/\"openshift-service-ca.crt\""
Aug 04 21:33:10.174013 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.173845    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a189ec\"/\"default-dockercfg-8rzqr\""
Aug 04 21:33:10.174904 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.174886    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a189ec\"/\"rate-test-model-a189ec-kserve-self-signed-certs\""
Aug 04 21:33:10.180476 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.180454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"]
Aug 04 21:33:10.302088 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.302058    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9df7b6-0a82-4022-b470-2c00a561c238-tls-certs\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.302274 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.302116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-dshm\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.302274 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.302143    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-home\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.302274 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.302178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-tmp-dir\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.302274 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.302198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-model-cache\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.302274 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.302218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gh6c\" (UniqueName: \"kubernetes.io/projected/2d9df7b6-0a82-4022-b470-2c00a561c238-kube-api-access-8gh6c\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.403312 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.403275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-model-cache\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.403312 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.403317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gh6c\" (UniqueName: \"kubernetes.io/projected/2d9df7b6-0a82-4022-b470-2c00a561c238-kube-api-access-8gh6c\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.403545 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.403393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9df7b6-0a82-4022-b470-2c00a561c238-tls-certs\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.403545 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.403450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-dshm\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.403545 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.403478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-home\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.403545 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.403501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-tmp-dir\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.403784 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.403757    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-model-cache\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.403845 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.403783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-home\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.403905 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.403876    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-tmp-dir\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.405688 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.405666    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-dshm\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.405996 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.405976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9df7b6-0a82-4022-b470-2c00a561c238-tls-certs\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.412534 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.412512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gh6c\" (UniqueName: \"kubernetes.io/projected/2d9df7b6-0a82-4022-b470-2c00a561c238-kube-api-access-8gh6c\") pod \"rate-test-model-a189ec-kserve-59df47d4dc-mbz68\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") " pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.480996 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.480965    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:10.615632 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.615602    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"]
Aug 04 21:33:10.618164 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:33:10.618132    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d9df7b6_0a82_4022_b470_2c00a561c238.slice/crio-daab0c3e8c077a74deccebd561ec840f9d36fb470c2627ef28b327ba09935587 WatchSource:0}: Error finding container daab0c3e8c077a74deccebd561ec840f9d36fb470c2627ef28b327ba09935587: Status 404 returned error can't find the container with id daab0c3e8c077a74deccebd561ec840f9d36fb470c2627ef28b327ba09935587
Aug 04 21:33:10.932003 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.931887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68" event={"ID":"2d9df7b6-0a82-4022-b470-2c00a561c238","Type":"ContainerStarted","Data":"0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09"}
Aug 04 21:33:10.932003 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.931938    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68" event={"ID":"2d9df7b6-0a82-4022-b470-2c00a561c238","Type":"ContainerStarted","Data":"daab0c3e8c077a74deccebd561ec840f9d36fb470c2627ef28b327ba09935587"}
Aug 04 21:33:10.961502 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:10.961451    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68" podStartSLOduration=0.961435839 podStartE2EDuration="961.435839ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:33:10.95962679 +0000 UTC m=+3165.711732770" watchObservedRunningTime="2026-08-04 21:33:10.961435839 +0000 UTC m=+3165.713541842"
Aug 04 21:33:12.276012 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.275961    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q" podUID="cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" containerName="main" containerID="cri-o://08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8" gracePeriod=45
Aug 04 21:33:12.528767 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.528700    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:33:12.627711 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.627679    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-home\") pod \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") "
Aug 04 21:33:12.627898 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.627728    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tls-certs\") pod \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") "
Aug 04 21:33:12.627898 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.627748    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tmp-dir\") pod \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") "
Aug 04 21:33:12.627898 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.627777    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-dshm\") pod \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") "
Aug 04 21:33:12.627898 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.627815    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-model-cache\") pod \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") "
Aug 04 21:33:12.627898 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.627884    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2lcz\" (UniqueName: \"kubernetes.io/projected/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-kube-api-access-k2lcz\") pod \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\" (UID: \"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa\") "
Aug 04 21:33:12.628188 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.628036    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-home" (OuterVolumeSpecName: "home") pod "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" (UID: "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:33:12.628188 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.628060    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" (UID: "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:33:12.628188 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.628123    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" (UID: "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:33:12.628383 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.628331    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:33:12.628440 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.628384    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:33:12.628440 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.628400    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:33:12.630018 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.629995    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-dshm" (OuterVolumeSpecName: "dshm") pod "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" (UID: "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:33:12.630198 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.630177    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" (UID: "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 21:33:12.630259 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.630175    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-kube-api-access-k2lcz" (OuterVolumeSpecName: "kube-api-access-k2lcz") pod "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" (UID: "cc426fd5-4e35-4a72-93ae-76c1ebeafaaa"). InnerVolumeSpecName "kube-api-access-k2lcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:33:12.729092 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.729050    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2lcz\" (UniqueName: \"kubernetes.io/projected/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-kube-api-access-k2lcz\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:33:12.729092 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.729090    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:33:12.729092 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.729102    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:33:12.852989 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.852891    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:33:12.945055 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.945017    2565 generic.go:356] "Generic (PLEG): container finished" podID="cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" containerID="08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8" exitCode=0
Aug 04 21:33:12.945248 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.945079    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"
Aug 04 21:33:12.945248 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.945102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q" event={"ID":"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa","Type":"ContainerDied","Data":"08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8"}
Aug 04 21:33:12.945248 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.945146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q" event={"ID":"cc426fd5-4e35-4a72-93ae-76c1ebeafaaa","Type":"ContainerDied","Data":"8d1c36fd1fdabc026ad0da232b500d618cc3ca58d1c05ec60791a577bee76495"}
Aug 04 21:33:12.945248 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.945163    2565 scope.go:117] "RemoveContainer" containerID="08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8"
Aug 04 21:33:12.954127 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.954110    2565 scope.go:117] "RemoveContainer" containerID="08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8"
Aug 04 21:33:12.954465 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:33:12.954441    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8\": container with ID starting with 08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8 not found: ID does not exist" containerID="08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8"
Aug 04 21:33:12.954553 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.954468    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8"} err="failed to get container status \"08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8\": rpc error: code = NotFound desc = could not find container \"08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8\": container with ID starting with 08ecf73d848dc4d1837ebba35c94a9fb7730db8884035c023ca98e7874368fa8 not found: ID does not exist"
Aug 04 21:33:12.966613 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.966590    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"]
Aug 04 21:33:12.970411 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:12.970390    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cd644a/sub-test-model-cd644a-kserve-846d6977f5-nmx8q"]
Aug 04 21:33:13.760509 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:13.760475    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" path="/var/lib/kubelet/pods/cc426fd5-4e35-4a72-93ae-76c1ebeafaaa/volumes"
Aug 04 21:33:20.481261 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:20.481217    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:20.481718 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:20.481306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:20.493752 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:20.493729    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:20.998928 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:20.998899    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:33:30.659601 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:33:30.659565    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:34:09.581489 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:09.581455    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:34:17.559768 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:17.559729    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:34:33.866756 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:33.866724    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:34:39.645636 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:39.645598    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"]
Aug 04 21:34:41.415758 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.415722    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7509-140509044235184"]
Aug 04 21:34:41.416248 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.416128    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" containerName="main"
Aug 04 21:34:41.416248 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.416149    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" containerName="main"
Aug 04 21:34:41.416248 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.416229    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc426fd5-4e35-4a72-93ae-76c1ebeafaaa" containerName="main"
Aug 04 21:34:41.420868 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.420834    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044235184"
Aug 04 21:34:41.424201 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.424174    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-k468w\""
Aug 04 21:34:41.432850 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.432816    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044235184"]
Aug 04 21:34:41.561837 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.561797    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbh4h\" (UniqueName: \"kubernetes.io/projected/4cf13ef1-9258-47ca-a432-dc6843151d3b-kube-api-access-zbh4h\") pod \"test-curl-7509-140509044235184\" (UID: \"4cf13ef1-9258-47ca-a432-dc6843151d3b\") " pod="opendatahub/test-curl-7509-140509044235184"
Aug 04 21:34:41.663186 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.663140    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbh4h\" (UniqueName: \"kubernetes.io/projected/4cf13ef1-9258-47ca-a432-dc6843151d3b-kube-api-access-zbh4h\") pod \"test-curl-7509-140509044235184\" (UID: \"4cf13ef1-9258-47ca-a432-dc6843151d3b\") " pod="opendatahub/test-curl-7509-140509044235184"
Aug 04 21:34:41.673333 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.673266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbh4h\" (UniqueName: \"kubernetes.io/projected/4cf13ef1-9258-47ca-a432-dc6843151d3b-kube-api-access-zbh4h\") pod \"test-curl-7509-140509044235184\" (UID: \"4cf13ef1-9258-47ca-a432-dc6843151d3b\") " pod="opendatahub/test-curl-7509-140509044235184"
Aug 04 21:34:41.732221 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.732176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044235184"
Aug 04 21:34:41.872322 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.872296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044235184"]
Aug 04 21:34:41.875053 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:34:41.875016    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cf13ef1_9258_47ca_a432_dc6843151d3b.slice/crio-9bcb1520a97750c884ae567e5cddb9886f6ebc3b2d5d2078b8899276bfd9ed8b WatchSource:0}: Error finding container 9bcb1520a97750c884ae567e5cddb9886f6ebc3b2d5d2078b8899276bfd9ed8b: Status 404 returned error can't find the container with id 9bcb1520a97750c884ae567e5cddb9886f6ebc3b2d5d2078b8899276bfd9ed8b
Aug 04 21:34:41.876393 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:41.876374    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 21:34:42.398102 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:42.398056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044235184" event={"ID":"4cf13ef1-9258-47ca-a432-dc6843151d3b","Type":"ContainerStarted","Data":"9bcb1520a97750c884ae567e5cddb9886f6ebc3b2d5d2078b8899276bfd9ed8b"}
Aug 04 21:34:42.556788 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:42.556758    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:34:43.404400 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:43.404338    2565 generic.go:356] "Generic (PLEG): container finished" podID="4cf13ef1-9258-47ca-a432-dc6843151d3b" containerID="aa696656ff29fedc44696f6028189e0aebe38ac3e5e30d24bce05557c59e6990" exitCode=0
Aug 04 21:34:43.404588 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:43.404437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044235184" event={"ID":"4cf13ef1-9258-47ca-a432-dc6843151d3b","Type":"ContainerDied","Data":"aa696656ff29fedc44696f6028189e0aebe38ac3e5e30d24bce05557c59e6990"}
Aug 04 21:34:44.549959 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.549930    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044235184"
Aug 04 21:34:44.590620 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.590593    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7509-140509044235184_4cf13ef1-9258-47ca-a432-dc6843151d3b/test-curl-7509-140509044235184/0.log"
Aug 04 21:34:44.591645 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.591621    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbh4h\" (UniqueName: \"kubernetes.io/projected/4cf13ef1-9258-47ca-a432-dc6843151d3b-kube-api-access-zbh4h\") pod \"4cf13ef1-9258-47ca-a432-dc6843151d3b\" (UID: \"4cf13ef1-9258-47ca-a432-dc6843151d3b\") "
Aug 04 21:34:44.593719 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.593693    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf13ef1-9258-47ca-a432-dc6843151d3b-kube-api-access-zbh4h" (OuterVolumeSpecName: "kube-api-access-zbh4h") pod "4cf13ef1-9258-47ca-a432-dc6843151d3b" (UID: "4cf13ef1-9258-47ca-a432-dc6843151d3b"). InnerVolumeSpecName "kube-api-access-zbh4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:34:44.619847 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.619818    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7509-140509044235184"]
Aug 04 21:34:44.621530 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.621509    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7509-140509044235184"]
Aug 04 21:34:44.692441 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.692379    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbh4h\" (UniqueName: \"kubernetes.io/projected/4cf13ef1-9258-47ca-a432-dc6843151d3b-kube-api-access-zbh4h\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:34:44.744333 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.744306    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7509-140509044232880"]
Aug 04 21:34:44.744723 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.744710    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf13ef1-9258-47ca-a432-dc6843151d3b" containerName="test-curl-7509-140509044235184"
Aug 04 21:34:44.744768 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.744725    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf13ef1-9258-47ca-a432-dc6843151d3b" containerName="test-curl-7509-140509044235184"
Aug 04 21:34:44.744808 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.744789    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf13ef1-9258-47ca-a432-dc6843151d3b" containerName="test-curl-7509-140509044235184"
Aug 04 21:34:44.747762 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.747748    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044232880"
Aug 04 21:34:44.756442 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.756418    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044232880"]
Aug 04 21:34:44.793595 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.793558    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8bms\" (UniqueName: \"kubernetes.io/projected/19311f41-ade5-4414-be58-ae208107b6e8-kube-api-access-r8bms\") pod \"test-curl-7509-140509044232880\" (UID: \"19311f41-ade5-4414-be58-ae208107b6e8\") " pod="opendatahub/test-curl-7509-140509044232880"
Aug 04 21:34:44.894412 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.894378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8bms\" (UniqueName: \"kubernetes.io/projected/19311f41-ade5-4414-be58-ae208107b6e8-kube-api-access-r8bms\") pod \"test-curl-7509-140509044232880\" (UID: \"19311f41-ade5-4414-be58-ae208107b6e8\") " pod="opendatahub/test-curl-7509-140509044232880"
Aug 04 21:34:44.903508 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:44.903486    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8bms\" (UniqueName: \"kubernetes.io/projected/19311f41-ade5-4414-be58-ae208107b6e8-kube-api-access-r8bms\") pod \"test-curl-7509-140509044232880\" (UID: \"19311f41-ade5-4414-be58-ae208107b6e8\") " pod="opendatahub/test-curl-7509-140509044232880"
Aug 04 21:34:45.057585 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:45.057551    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044232880"
Aug 04 21:34:45.192532 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:45.192508    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044232880"]
Aug 04 21:34:45.194657 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:34:45.194626    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19311f41_ade5_4414_be58_ae208107b6e8.slice/crio-c8a414ddc93d6860e1104ac9bcb7b193f940c574857246310600cd002defc283 WatchSource:0}: Error finding container c8a414ddc93d6860e1104ac9bcb7b193f940c574857246310600cd002defc283: Status 404 returned error can't find the container with id c8a414ddc93d6860e1104ac9bcb7b193f940c574857246310600cd002defc283
Aug 04 21:34:45.414825 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:45.414736    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044235184"
Aug 04 21:34:45.414825 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:45.414771    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bcb1520a97750c884ae567e5cddb9886f6ebc3b2d5d2078b8899276bfd9ed8b"
Aug 04 21:34:45.415838 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:45.415811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044232880" event={"ID":"19311f41-ade5-4414-be58-ae208107b6e8","Type":"ContainerStarted","Data":"c8a414ddc93d6860e1104ac9bcb7b193f940c574857246310600cd002defc283"}
Aug 04 21:34:45.425616 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:34:45.425593    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7509-140509044235184\" is forbidden: User \"system:node:ip-10-0-140-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-103.ec2.internal' and this object" podUID="4cf13ef1-9258-47ca-a432-dc6843151d3b" pod="opendatahub/test-curl-7509-140509044235184"
Aug 04 21:34:45.758940 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:34:45.758908    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7509-140509044235184\" is forbidden: User \"system:node:ip-10-0-140-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-103.ec2.internal' and this object" podUID="4cf13ef1-9258-47ca-a432-dc6843151d3b" pod="opendatahub/test-curl-7509-140509044235184"
Aug 04 21:34:45.760297 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:45.760275    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf13ef1-9258-47ca-a432-dc6843151d3b" path="/var/lib/kubelet/pods/4cf13ef1-9258-47ca-a432-dc6843151d3b/volumes"
Aug 04 21:34:46.422173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:46.422141    2565 generic.go:356] "Generic (PLEG): container finished" podID="19311f41-ade5-4414-be58-ae208107b6e8" containerID="d1946b6ebdfe9d9c18fffac52c9c008c2baa7a4b8fdb11dafd0f3d29e33f2785" exitCode=0
Aug 04 21:34:46.422334 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:46.422208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044232880" event={"ID":"19311f41-ade5-4414-be58-ae208107b6e8","Type":"ContainerDied","Data":"d1946b6ebdfe9d9c18fffac52c9c008c2baa7a4b8fdb11dafd0f3d29e33f2785"}
Aug 04 21:34:47.579853 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.579823    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044232880"
Aug 04 21:34:47.602190 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.602162    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7509-140509044232880_19311f41-ade5-4414-be58-ae208107b6e8/test-curl-7509-140509044232880/0.log"
Aug 04 21:34:47.619710 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.619680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8bms\" (UniqueName: \"kubernetes.io/projected/19311f41-ade5-4414-be58-ae208107b6e8-kube-api-access-r8bms\") pod \"19311f41-ade5-4414-be58-ae208107b6e8\" (UID: \"19311f41-ade5-4414-be58-ae208107b6e8\") "
Aug 04 21:34:47.621869 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.621843    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19311f41-ade5-4414-be58-ae208107b6e8-kube-api-access-r8bms" (OuterVolumeSpecName: "kube-api-access-r8bms") pod "19311f41-ade5-4414-be58-ae208107b6e8" (UID: "19311f41-ade5-4414-be58-ae208107b6e8"). InnerVolumeSpecName "kube-api-access-r8bms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:34:47.631493 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.631466    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7509-140509044232880"]
Aug 04 21:34:47.637227 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.637206    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7509-140509044232880"]
Aug 04 21:34:47.721228 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.721205    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8bms\" (UniqueName: \"kubernetes.io/projected/19311f41-ade5-4414-be58-ae208107b6e8-kube-api-access-r8bms\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:34:47.761080 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.761048    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19311f41-ade5-4414-be58-ae208107b6e8" path="/var/lib/kubelet/pods/19311f41-ade5-4414-be58-ae208107b6e8/volumes"
Aug 04 21:34:47.801469 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.801434    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7509-140509044383008"]
Aug 04 21:34:47.801861 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.801848    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19311f41-ade5-4414-be58-ae208107b6e8" containerName="test-curl-7509-140509044232880"
Aug 04 21:34:47.801910 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.801864    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19311f41-ade5-4414-be58-ae208107b6e8" containerName="test-curl-7509-140509044232880"
Aug 04 21:34:47.801946 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.801941    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19311f41-ade5-4414-be58-ae208107b6e8" containerName="test-curl-7509-140509044232880"
Aug 04 21:34:47.805073 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.805058    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:47.816049 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.816026    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044383008"]
Aug 04 21:34:47.821646 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.821624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbcr\" (UniqueName: \"kubernetes.io/projected/e306a2bd-2464-433d-929d-6528838679ad-kube-api-access-6nbcr\") pod \"test-curl-7509-140509044383008\" (UID: \"e306a2bd-2464-433d-929d-6528838679ad\") " pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:47.922915 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.922874    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbcr\" (UniqueName: \"kubernetes.io/projected/e306a2bd-2464-433d-929d-6528838679ad-kube-api-access-6nbcr\") pod \"test-curl-7509-140509044383008\" (UID: \"e306a2bd-2464-433d-929d-6528838679ad\") " pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:47.931291 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:47.931259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbcr\" (UniqueName: \"kubernetes.io/projected/e306a2bd-2464-433d-929d-6528838679ad-kube-api-access-6nbcr\") pod \"test-curl-7509-140509044383008\" (UID: \"e306a2bd-2464-433d-929d-6528838679ad\") " pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:48.113996 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:48.113897    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:48.256924 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:48.256900    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044383008"]
Aug 04 21:34:48.258370 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:34:48.258322    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode306a2bd_2464_433d_929d_6528838679ad.slice/crio-1706b6ace07cf4cf89feed25dfd58063fdc576a1218624e8d2408840bc07de28 WatchSource:0}: Error finding container 1706b6ace07cf4cf89feed25dfd58063fdc576a1218624e8d2408840bc07de28: Status 404 returned error can't find the container with id 1706b6ace07cf4cf89feed25dfd58063fdc576a1218624e8d2408840bc07de28
Aug 04 21:34:48.434730 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:48.434645    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044232880"
Aug 04 21:34:48.434730 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:48.434686    2565 scope.go:117] "RemoveContainer" containerID="d1946b6ebdfe9d9c18fffac52c9c008c2baa7a4b8fdb11dafd0f3d29e33f2785"
Aug 04 21:34:48.436168 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:48.435904    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044383008" event={"ID":"e306a2bd-2464-433d-929d-6528838679ad","Type":"ContainerStarted","Data":"1706b6ace07cf4cf89feed25dfd58063fdc576a1218624e8d2408840bc07de28"}
Aug 04 21:34:49.444145 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:49.444104    2565 generic.go:356] "Generic (PLEG): container finished" podID="e306a2bd-2464-433d-929d-6528838679ad" containerID="eff61879c7d856e935daa38db04e692b8600fdf8944a73fbee2def22593e7ca8" exitCode=0
Aug 04 21:34:49.444145 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:49.444149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044383008" event={"ID":"e306a2bd-2464-433d-929d-6528838679ad","Type":"ContainerDied","Data":"eff61879c7d856e935daa38db04e692b8600fdf8944a73fbee2def22593e7ca8"}
Aug 04 21:34:50.587323 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.587294    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:50.609114 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.609093    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7509-140509044383008_e306a2bd-2464-433d-929d-6528838679ad/test-curl-7509-140509044383008/0.log"
Aug 04 21:34:50.637808 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.637780    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7509-140509044383008"]
Aug 04 21:34:50.640124 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.640099    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7509-140509044383008"]
Aug 04 21:34:50.650476 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.650458    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nbcr\" (UniqueName: \"kubernetes.io/projected/e306a2bd-2464-433d-929d-6528838679ad-kube-api-access-6nbcr\") pod \"e306a2bd-2464-433d-929d-6528838679ad\" (UID: \"e306a2bd-2464-433d-929d-6528838679ad\") "
Aug 04 21:34:50.652454 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.652433    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e306a2bd-2464-433d-929d-6528838679ad-kube-api-access-6nbcr" (OuterVolumeSpecName: "kube-api-access-6nbcr") pod "e306a2bd-2464-433d-929d-6528838679ad" (UID: "e306a2bd-2464-433d-929d-6528838679ad"). InnerVolumeSpecName "kube-api-access-6nbcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:34:50.751972 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.751934    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbcr\" (UniqueName: \"kubernetes.io/projected/e306a2bd-2464-433d-929d-6528838679ad-kube-api-access-6nbcr\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:34:50.767840 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.767813    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7509-140509044383008"]
Aug 04 21:34:50.768231 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.768216    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e306a2bd-2464-433d-929d-6528838679ad" containerName="test-curl-7509-140509044383008"
Aug 04 21:34:50.768277 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.768232    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e306a2bd-2464-433d-929d-6528838679ad" containerName="test-curl-7509-140509044383008"
Aug 04 21:34:50.768313 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.768297    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e306a2bd-2464-433d-929d-6528838679ad" containerName="test-curl-7509-140509044383008"
Aug 04 21:34:50.771326 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.771310    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:50.790593 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.790559    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044383008"]
Aug 04 21:34:50.852827 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.852796    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pphs\" (UniqueName: \"kubernetes.io/projected/a10550df-0a05-4d8c-a9ea-154bc19260cb-kube-api-access-7pphs\") pod \"test-curl-7509-140509044383008\" (UID: \"a10550df-0a05-4d8c-a9ea-154bc19260cb\") " pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:50.953712 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.953676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pphs\" (UniqueName: \"kubernetes.io/projected/a10550df-0a05-4d8c-a9ea-154bc19260cb-kube-api-access-7pphs\") pod \"test-curl-7509-140509044383008\" (UID: \"a10550df-0a05-4d8c-a9ea-154bc19260cb\") " pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:50.966582 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:50.966554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pphs\" (UniqueName: \"kubernetes.io/projected/a10550df-0a05-4d8c-a9ea-154bc19260cb-kube-api-access-7pphs\") pod \"test-curl-7509-140509044383008\" (UID: \"a10550df-0a05-4d8c-a9ea-154bc19260cb\") " pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:51.079587 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:51.079503    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:51.215828 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:51.215803    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044383008"]
Aug 04 21:34:51.218078 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:34:51.218047    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda10550df_0a05_4d8c_a9ea_154bc19260cb.slice/crio-50ff740b20a569eccf79b1520f66eba4f7fc7958dcdbf6f21f291c27fdc46411 WatchSource:0}: Error finding container 50ff740b20a569eccf79b1520f66eba4f7fc7958dcdbf6f21f291c27fdc46411: Status 404 returned error can't find the container with id 50ff740b20a569eccf79b1520f66eba4f7fc7958dcdbf6f21f291c27fdc46411
Aug 04 21:34:51.454932 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:51.454841    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1706b6ace07cf4cf89feed25dfd58063fdc576a1218624e8d2408840bc07de28"
Aug 04 21:34:51.454932 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:51.454851    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:51.456159 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:51.456122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044383008" event={"ID":"a10550df-0a05-4d8c-a9ea-154bc19260cb","Type":"ContainerStarted","Data":"50ff740b20a569eccf79b1520f66eba4f7fc7958dcdbf6f21f291c27fdc46411"}
Aug 04 21:34:51.467020 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:51.466987    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7509-140509044383008" oldPodUID="e306a2bd-2464-433d-929d-6528838679ad" podUID="a10550df-0a05-4d8c-a9ea-154bc19260cb"
Aug 04 21:34:51.761193 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:51.761167    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e306a2bd-2464-433d-929d-6528838679ad" path="/var/lib/kubelet/pods/e306a2bd-2464-433d-929d-6528838679ad/volumes"
Aug 04 21:34:52.462852 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:52.462816    2565 generic.go:356] "Generic (PLEG): container finished" podID="a10550df-0a05-4d8c-a9ea-154bc19260cb" containerID="557641bc6dd331025b108ba2faabb7b8e759f79c1464bcc8867462f3fdf7aeb7" exitCode=0
Aug 04 21:34:52.463000 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:52.462901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044383008" event={"ID":"a10550df-0a05-4d8c-a9ea-154bc19260cb","Type":"ContainerDied","Data":"557641bc6dd331025b108ba2faabb7b8e759f79c1464bcc8867462f3fdf7aeb7"}
Aug 04 21:34:53.600664 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.600643    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:53.627301 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.627279    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7509-140509044383008_a10550df-0a05-4d8c-a9ea-154bc19260cb/test-curl-7509-140509044383008/0.log"
Aug 04 21:34:53.664611 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.664580    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7509-140509044383008"]
Aug 04 21:34:53.670718 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.670696    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7509-140509044383008"]
Aug 04 21:34:53.682646 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.682625    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pphs\" (UniqueName: \"kubernetes.io/projected/a10550df-0a05-4d8c-a9ea-154bc19260cb-kube-api-access-7pphs\") pod \"a10550df-0a05-4d8c-a9ea-154bc19260cb\" (UID: \"a10550df-0a05-4d8c-a9ea-154bc19260cb\") "
Aug 04 21:34:53.684679 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.684653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10550df-0a05-4d8c-a9ea-154bc19260cb-kube-api-access-7pphs" (OuterVolumeSpecName: "kube-api-access-7pphs") pod "a10550df-0a05-4d8c-a9ea-154bc19260cb" (UID: "a10550df-0a05-4d8c-a9ea-154bc19260cb"). InnerVolumeSpecName "kube-api-access-7pphs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:34:53.761172 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.761109    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10550df-0a05-4d8c-a9ea-154bc19260cb" path="/var/lib/kubelet/pods/a10550df-0a05-4d8c-a9ea-154bc19260cb/volumes"
Aug 04 21:34:53.783671 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.783649    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pphs\" (UniqueName: \"kubernetes.io/projected/a10550df-0a05-4d8c-a9ea-154bc19260cb-kube-api-access-7pphs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:34:53.799066 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.799041    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7509-140509044273488"]
Aug 04 21:34:53.799424 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.799412    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10550df-0a05-4d8c-a9ea-154bc19260cb" containerName="test-curl-7509-140509044383008"
Aug 04 21:34:53.799464 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.799425    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10550df-0a05-4d8c-a9ea-154bc19260cb" containerName="test-curl-7509-140509044383008"
Aug 04 21:34:53.799508 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.799501    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10550df-0a05-4d8c-a9ea-154bc19260cb" containerName="test-curl-7509-140509044383008"
Aug 04 21:34:53.802496 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.802482    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:53.816876 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.816853    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044273488"]
Aug 04 21:34:53.884378 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.884326    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk89p\" (UniqueName: \"kubernetes.io/projected/b8b850cb-40ab-4902-8e5e-1dcf4308894c-kube-api-access-rk89p\") pod \"test-curl-7509-140509044273488\" (UID: \"b8b850cb-40ab-4902-8e5e-1dcf4308894c\") " pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:53.985271 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.985240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk89p\" (UniqueName: \"kubernetes.io/projected/b8b850cb-40ab-4902-8e5e-1dcf4308894c-kube-api-access-rk89p\") pod \"test-curl-7509-140509044273488\" (UID: \"b8b850cb-40ab-4902-8e5e-1dcf4308894c\") " pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:53.994177 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:53.994151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk89p\" (UniqueName: \"kubernetes.io/projected/b8b850cb-40ab-4902-8e5e-1dcf4308894c-kube-api-access-rk89p\") pod \"test-curl-7509-140509044273488\" (UID: \"b8b850cb-40ab-4902-8e5e-1dcf4308894c\") " pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:54.111539 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.111454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:54.251478 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.251451    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044273488"]
Aug 04 21:34:54.253888 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:34:54.253863    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8b850cb_40ab_4902_8e5e_1dcf4308894c.slice/crio-30cd4ba2e4dfe3d1ddff2e065b59b0ae369afc6fbe913b3029cfca021817fd1a WatchSource:0}: Error finding container 30cd4ba2e4dfe3d1ddff2e065b59b0ae369afc6fbe913b3029cfca021817fd1a: Status 404 returned error can't find the container with id 30cd4ba2e4dfe3d1ddff2e065b59b0ae369afc6fbe913b3029cfca021817fd1a
Aug 04 21:34:54.475415 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.475389    2565 scope.go:117] "RemoveContainer" containerID="557641bc6dd331025b108ba2faabb7b8e759f79c1464bcc8867462f3fdf7aeb7"
Aug 04 21:34:54.475583 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.475390    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044383008"
Aug 04 21:34:54.476735 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.476589    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044273488" event={"ID":"b8b850cb-40ab-4902-8e5e-1dcf4308894c","Type":"ContainerStarted","Data":"30cd4ba2e4dfe3d1ddff2e065b59b0ae369afc6fbe913b3029cfca021817fd1a"}
Aug 04 21:34:54.668744 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.668685    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68" podUID="2d9df7b6-0a82-4022-b470-2c00a561c238" containerName="main" containerID="cri-o://0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09" gracePeriod=45
Aug 04 21:34:54.920686 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.920663    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:34:54.994250 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994219    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-home\") pod \"2d9df7b6-0a82-4022-b470-2c00a561c238\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") "
Aug 04 21:34:54.994250 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-tmp-dir\") pod \"2d9df7b6-0a82-4022-b470-2c00a561c238\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") "
Aug 04 21:34:54.994502 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994278    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-model-cache\") pod \"2d9df7b6-0a82-4022-b470-2c00a561c238\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") "
Aug 04 21:34:54.994502 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994304    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-dshm\") pod \"2d9df7b6-0a82-4022-b470-2c00a561c238\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") "
Aug 04 21:34:54.994502 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994396    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gh6c\" (UniqueName: \"kubernetes.io/projected/2d9df7b6-0a82-4022-b470-2c00a561c238-kube-api-access-8gh6c\") pod \"2d9df7b6-0a82-4022-b470-2c00a561c238\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") "
Aug 04 21:34:54.994502 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994462    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9df7b6-0a82-4022-b470-2c00a561c238-tls-certs\") pod \"2d9df7b6-0a82-4022-b470-2c00a561c238\" (UID: \"2d9df7b6-0a82-4022-b470-2c00a561c238\") "
Aug 04 21:34:54.994755 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994552    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d9df7b6-0a82-4022-b470-2c00a561c238" (UID: "2d9df7b6-0a82-4022-b470-2c00a561c238"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:34:54.994755 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994567    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-home" (OuterVolumeSpecName: "home") pod "2d9df7b6-0a82-4022-b470-2c00a561c238" (UID: "2d9df7b6-0a82-4022-b470-2c00a561c238"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:34:54.994755 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994578    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d9df7b6-0a82-4022-b470-2c00a561c238" (UID: "2d9df7b6-0a82-4022-b470-2c00a561c238"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:34:54.994901 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994757    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:34:54.994901 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994776    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:34:54.994901 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.994789    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:34:54.996509 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.996484    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-dshm" (OuterVolumeSpecName: "dshm") pod "2d9df7b6-0a82-4022-b470-2c00a561c238" (UID: "2d9df7b6-0a82-4022-b470-2c00a561c238"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:34:54.996597 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.996564    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9df7b6-0a82-4022-b470-2c00a561c238-kube-api-access-8gh6c" (OuterVolumeSpecName: "kube-api-access-8gh6c") pod "2d9df7b6-0a82-4022-b470-2c00a561c238" (UID: "2d9df7b6-0a82-4022-b470-2c00a561c238"). InnerVolumeSpecName "kube-api-access-8gh6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:34:54.996640 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:54.996588    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9df7b6-0a82-4022-b470-2c00a561c238-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d9df7b6-0a82-4022-b470-2c00a561c238" (UID: "2d9df7b6-0a82-4022-b470-2c00a561c238"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 21:34:55.095800 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.095713    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gh6c\" (UniqueName: \"kubernetes.io/projected/2d9df7b6-0a82-4022-b470-2c00a561c238-kube-api-access-8gh6c\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:34:55.095800 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.095743    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9df7b6-0a82-4022-b470-2c00a561c238-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:34:55.095800 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.095755    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9df7b6-0a82-4022-b470-2c00a561c238-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:34:55.483260 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.483229    2565 generic.go:356] "Generic (PLEG): container finished" podID="b8b850cb-40ab-4902-8e5e-1dcf4308894c" containerID="6feb07bc990a7fc8406048a5b52be412da13024e4d12a1c5f01ee3b3c7eb2374" exitCode=0
Aug 04 21:34:55.483457 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.483306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044273488" event={"ID":"b8b850cb-40ab-4902-8e5e-1dcf4308894c","Type":"ContainerDied","Data":"6feb07bc990a7fc8406048a5b52be412da13024e4d12a1c5f01ee3b3c7eb2374"}
Aug 04 21:34:55.484603 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.484579    2565 generic.go:356] "Generic (PLEG): container finished" podID="2d9df7b6-0a82-4022-b470-2c00a561c238" containerID="0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09" exitCode=0
Aug 04 21:34:55.484691 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.484615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68" event={"ID":"2d9df7b6-0a82-4022-b470-2c00a561c238","Type":"ContainerDied","Data":"0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09"}
Aug 04 21:34:55.484691 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.484650    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68" event={"ID":"2d9df7b6-0a82-4022-b470-2c00a561c238","Type":"ContainerDied","Data":"daab0c3e8c077a74deccebd561ec840f9d36fb470c2627ef28b327ba09935587"}
Aug 04 21:34:55.484691 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.484670    2565 scope.go:117] "RemoveContainer" containerID="0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09"
Aug 04 21:34:55.484691 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.484622    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"
Aug 04 21:34:55.493179 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.493163    2565 scope.go:117] "RemoveContainer" containerID="0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09"
Aug 04 21:34:55.493478 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:34:55.493455    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09\": container with ID starting with 0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09 not found: ID does not exist" containerID="0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09"
Aug 04 21:34:55.493542 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.493490    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09"} err="failed to get container status \"0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09\": rpc error: code = NotFound desc = could not find container \"0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09\": container with ID starting with 0f706652767a01aa2910bafbea0d51d94f975d386e5e79e3e6a8fac969e76b09 not found: ID does not exist"
Aug 04 21:34:55.522794 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.522763    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"]
Aug 04 21:34:55.527207 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.527185    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a189ec/rate-test-model-a189ec-kserve-59df47d4dc-mbz68"]
Aug 04 21:34:55.761121 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:55.761029    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9df7b6-0a82-4022-b470-2c00a561c238" path="/var/lib/kubelet/pods/2d9df7b6-0a82-4022-b470-2c00a561c238/volumes"
Aug 04 21:34:56.631009 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.630986    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:56.662123 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.662099    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7509-140509044273488_b8b850cb-40ab-4902-8e5e-1dcf4308894c/test-curl-7509-140509044273488/0.log"
Aug 04 21:34:56.685955 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.685925    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7509-140509044273488"]
Aug 04 21:34:56.691827 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.691807    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7509-140509044273488"]
Aug 04 21:34:56.712056 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.712035    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk89p\" (UniqueName: \"kubernetes.io/projected/b8b850cb-40ab-4902-8e5e-1dcf4308894c-kube-api-access-rk89p\") pod \"b8b850cb-40ab-4902-8e5e-1dcf4308894c\" (UID: \"b8b850cb-40ab-4902-8e5e-1dcf4308894c\") "
Aug 04 21:34:56.714020 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.713999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b850cb-40ab-4902-8e5e-1dcf4308894c-kube-api-access-rk89p" (OuterVolumeSpecName: "kube-api-access-rk89p") pod "b8b850cb-40ab-4902-8e5e-1dcf4308894c" (UID: "b8b850cb-40ab-4902-8e5e-1dcf4308894c"). InnerVolumeSpecName "kube-api-access-rk89p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:34:56.813411 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.813301    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk89p\" (UniqueName: \"kubernetes.io/projected/b8b850cb-40ab-4902-8e5e-1dcf4308894c-kube-api-access-rk89p\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:34:56.818548 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.818522    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7509-140509044273488"]
Aug 04 21:34:56.818913 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.818902    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b850cb-40ab-4902-8e5e-1dcf4308894c" containerName="test-curl-7509-140509044273488"
Aug 04 21:34:56.818957 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.818915    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b850cb-40ab-4902-8e5e-1dcf4308894c" containerName="test-curl-7509-140509044273488"
Aug 04 21:34:56.818957 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.818933    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9df7b6-0a82-4022-b470-2c00a561c238" containerName="main"
Aug 04 21:34:56.818957 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.818939    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9df7b6-0a82-4022-b470-2c00a561c238" containerName="main"
Aug 04 21:34:56.819052 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.818992    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b850cb-40ab-4902-8e5e-1dcf4308894c" containerName="test-curl-7509-140509044273488"
Aug 04 21:34:56.819052 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.819000    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9df7b6-0a82-4022-b470-2c00a561c238" containerName="main"
Aug 04 21:34:56.822011 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.821997    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:56.830380 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.830324    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044273488"]
Aug 04 21:34:56.914661 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:56.914632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgnkn\" (UniqueName: \"kubernetes.io/projected/a3d1cfd5-4299-4df0-86d4-9f1f8dd50922-kube-api-access-dgnkn\") pod \"test-curl-7509-140509044273488\" (UID: \"a3d1cfd5-4299-4df0-86d4-9f1f8dd50922\") " pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:57.015276 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:57.015244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgnkn\" (UniqueName: \"kubernetes.io/projected/a3d1cfd5-4299-4df0-86d4-9f1f8dd50922-kube-api-access-dgnkn\") pod \"test-curl-7509-140509044273488\" (UID: \"a3d1cfd5-4299-4df0-86d4-9f1f8dd50922\") " pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:57.024417 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:57.024388    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgnkn\" (UniqueName: \"kubernetes.io/projected/a3d1cfd5-4299-4df0-86d4-9f1f8dd50922-kube-api-access-dgnkn\") pod \"test-curl-7509-140509044273488\" (UID: \"a3d1cfd5-4299-4df0-86d4-9f1f8dd50922\") " pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:57.132168 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:57.132094    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:57.486968 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:57.486935    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-140509044273488"]
Aug 04 21:34:57.491492 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:34:57.491463    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3d1cfd5_4299_4df0_86d4_9f1f8dd50922.slice/crio-239fc64da791757b64358dd368d41ebfa4af3e67ea0a47e6a6c8857f099451cf WatchSource:0}: Error finding container 239fc64da791757b64358dd368d41ebfa4af3e67ea0a47e6a6c8857f099451cf: Status 404 returned error can't find the container with id 239fc64da791757b64358dd368d41ebfa4af3e67ea0a47e6a6c8857f099451cf
Aug 04 21:34:57.497417 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:57.497398    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:57.497490 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:57.497437    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30cd4ba2e4dfe3d1ddff2e065b59b0ae369afc6fbe913b3029cfca021817fd1a"
Aug 04 21:34:57.510261 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:57.510233    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7509-140509044273488" oldPodUID="b8b850cb-40ab-4902-8e5e-1dcf4308894c" podUID="a3d1cfd5-4299-4df0-86d4-9f1f8dd50922"
Aug 04 21:34:57.761173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:57.761085    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b850cb-40ab-4902-8e5e-1dcf4308894c" path="/var/lib/kubelet/pods/b8b850cb-40ab-4902-8e5e-1dcf4308894c/volumes"
Aug 04 21:34:58.504244 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:58.504207    2565 generic.go:356] "Generic (PLEG): container finished" podID="a3d1cfd5-4299-4df0-86d4-9f1f8dd50922" containerID="35c443b5eaa00b53ebd1ebcfcb20390dbd6a273cba78f4e0c4c9b93a9b2c97e4" exitCode=0
Aug 04 21:34:58.504651 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:58.504289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044273488" event={"ID":"a3d1cfd5-4299-4df0-86d4-9f1f8dd50922","Type":"ContainerDied","Data":"35c443b5eaa00b53ebd1ebcfcb20390dbd6a273cba78f4e0c4c9b93a9b2c97e4"}
Aug 04 21:34:58.504651 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:58.504325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-140509044273488" event={"ID":"a3d1cfd5-4299-4df0-86d4-9f1f8dd50922","Type":"ContainerStarted","Data":"239fc64da791757b64358dd368d41ebfa4af3e67ea0a47e6a6c8857f099451cf"}
Aug 04 21:34:59.649910 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:59.649887    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:34:59.675237 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:59.675208    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7509-140509044273488_a3d1cfd5-4299-4df0-86d4-9f1f8dd50922/test-curl-7509-140509044273488/0.log"
Aug 04 21:34:59.703933 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:59.703904    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7509-140509044273488"]
Aug 04 21:34:59.707967 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:59.707943    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7509-140509044273488"]
Aug 04 21:34:59.741608 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:59.741588    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgnkn\" (UniqueName: \"kubernetes.io/projected/a3d1cfd5-4299-4df0-86d4-9f1f8dd50922-kube-api-access-dgnkn\") pod \"a3d1cfd5-4299-4df0-86d4-9f1f8dd50922\" (UID: \"a3d1cfd5-4299-4df0-86d4-9f1f8dd50922\") "
Aug 04 21:34:59.743611 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:59.743590    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d1cfd5-4299-4df0-86d4-9f1f8dd50922-kube-api-access-dgnkn" (OuterVolumeSpecName: "kube-api-access-dgnkn") pod "a3d1cfd5-4299-4df0-86d4-9f1f8dd50922" (UID: "a3d1cfd5-4299-4df0-86d4-9f1f8dd50922"). InnerVolumeSpecName "kube-api-access-dgnkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:34:59.760781 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:59.760725    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d1cfd5-4299-4df0-86d4-9f1f8dd50922" path="/var/lib/kubelet/pods/a3d1cfd5-4299-4df0-86d4-9f1f8dd50922/volumes"
Aug 04 21:34:59.843024 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:34:59.842992    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgnkn\" (UniqueName: \"kubernetes.io/projected/a3d1cfd5-4299-4df0-86d4-9f1f8dd50922-kube-api-access-dgnkn\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:35:00.515395 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:00.515361    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-140509044273488"
Aug 04 21:35:00.515571 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:00.515318    2565 scope.go:117] "RemoveContainer" containerID="35c443b5eaa00b53ebd1ebcfcb20390dbd6a273cba78f4e0c4c9b93a9b2c97e4"
Aug 04 21:35:04.972332 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:04.972295    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"]
Aug 04 21:35:04.972735 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:04.972695    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d1cfd5-4299-4df0-86d4-9f1f8dd50922" containerName="test-curl-7509-140509044273488"
Aug 04 21:35:04.972735 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:04.972707    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d1cfd5-4299-4df0-86d4-9f1f8dd50922" containerName="test-curl-7509-140509044273488"
Aug 04 21:35:04.972832 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:04.972786    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d1cfd5-4299-4df0-86d4-9f1f8dd50922" containerName="test-curl-7509-140509044273488"
Aug 04 21:35:04.977592 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:04.977572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:35:04.980280 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:04.980262    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e783f6-dockercfg-vm9g7\""
Aug 04 21:35:04.980515 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:04.980494    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e783f6\""
Aug 04 21:35:04.988508 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:04.988487    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"]
Aug 04 21:35:04.999779 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:04.999753    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"]
Aug 04 21:35:05.003463 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.003433    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:35:05.017111 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.017090    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"]
Aug 04 21:35:05.094985 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.094956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909af80f-7fcd-4da0-8e07-b1546085f659-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm\" (UID: \"909af80f-7fcd-4da0-8e07-b1546085f659\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:35:05.095170 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.094999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqczd\" (UniqueName: \"kubernetes.io/projected/909af80f-7fcd-4da0-8e07-b1546085f659-kube-api-access-cqczd\") pod \"payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm\" (UID: \"909af80f-7fcd-4da0-8e07-b1546085f659\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:35:05.095170 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.095107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7f4adc-f784-474f-acd9-11cd81d4a166-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr\" (UID: \"9d7f4adc-f784-474f-acd9-11cd81d4a166\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:35:05.095170 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.095148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9bp\" (UniqueName: \"kubernetes.io/projected/9d7f4adc-f784-474f-acd9-11cd81d4a166-kube-api-access-jg9bp\") pod \"payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr\" (UID: \"9d7f4adc-f784-474f-acd9-11cd81d4a166\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:35:05.196318 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.196278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909af80f-7fcd-4da0-8e07-b1546085f659-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm\" (UID: \"909af80f-7fcd-4da0-8e07-b1546085f659\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:35:05.196318 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.196320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqczd\" (UniqueName: \"kubernetes.io/projected/909af80f-7fcd-4da0-8e07-b1546085f659-kube-api-access-cqczd\") pod \"payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm\" (UID: \"909af80f-7fcd-4da0-8e07-b1546085f659\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:35:05.196553 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.196377    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7f4adc-f784-474f-acd9-11cd81d4a166-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr\" (UID: \"9d7f4adc-f784-474f-acd9-11cd81d4a166\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:35:05.196553 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.196396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9bp\" (UniqueName: \"kubernetes.io/projected/9d7f4adc-f784-474f-acd9-11cd81d4a166-kube-api-access-jg9bp\") pod \"payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr\" (UID: \"9d7f4adc-f784-474f-acd9-11cd81d4a166\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:35:05.197211 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.197188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7f4adc-f784-474f-acd9-11cd81d4a166-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr\" (UID: \"9d7f4adc-f784-474f-acd9-11cd81d4a166\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:35:05.197251 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.197208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909af80f-7fcd-4da0-8e07-b1546085f659-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm\" (UID: \"909af80f-7fcd-4da0-8e07-b1546085f659\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:35:05.204945 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.204920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9bp\" (UniqueName: \"kubernetes.io/projected/9d7f4adc-f784-474f-acd9-11cd81d4a166-kube-api-access-jg9bp\") pod \"payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr\" (UID: \"9d7f4adc-f784-474f-acd9-11cd81d4a166\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:35:05.205060 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.205035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqczd\" (UniqueName: \"kubernetes.io/projected/909af80f-7fcd-4da0-8e07-b1546085f659-kube-api-access-cqczd\") pod \"payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm\" (UID: \"909af80f-7fcd-4da0-8e07-b1546085f659\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:35:05.289175 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.289082    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:35:05.314092 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.314060    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:35:05.439476 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.439443    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"]
Aug 04 21:35:05.441576 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:35:05.441542    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909af80f_7fcd_4da0_8e07_b1546085f659.slice/crio-89372ff1775d3c4b364435a8bc8ab83a52e4b17613239762b0f25b4fb5248528 WatchSource:0}: Error finding container 89372ff1775d3c4b364435a8bc8ab83a52e4b17613239762b0f25b4fb5248528: Status 404 returned error can't find the container with id 89372ff1775d3c4b364435a8bc8ab83a52e4b17613239762b0f25b4fb5248528
Aug 04 21:35:05.542115 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.542015    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm" event={"ID":"909af80f-7fcd-4da0-8e07-b1546085f659","Type":"ContainerStarted","Data":"8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a"}
Aug 04 21:35:05.542115 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.542066    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm" event={"ID":"909af80f-7fcd-4da0-8e07-b1546085f659","Type":"ContainerStarted","Data":"89372ff1775d3c4b364435a8bc8ab83a52e4b17613239762b0f25b4fb5248528"}
Aug 04 21:35:05.542366 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.542134    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:35:05.561829 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.561785    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm" podStartSLOduration=1.561768921 podStartE2EDuration="1.561768921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:35:05.561482495 +0000 UTC m=+3280.313588472" watchObservedRunningTime="2026-08-04 21:35:05.561768921 +0000 UTC m=+3280.313874898"
Aug 04 21:35:05.674772 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:05.674710    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"]
Aug 04 21:35:05.677141 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:35:05.677112    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7f4adc_f784_474f_acd9_11cd81d4a166.slice/crio-835e3230d03e6ee5a3c1ace364a1fc957abe61303b394700ea918bda28187713 WatchSource:0}: Error finding container 835e3230d03e6ee5a3c1ace364a1fc957abe61303b394700ea918bda28187713: Status 404 returned error can't find the container with id 835e3230d03e6ee5a3c1ace364a1fc957abe61303b394700ea918bda28187713
Aug 04 21:35:06.549039 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:06.549000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr" event={"ID":"9d7f4adc-f784-474f-acd9-11cd81d4a166","Type":"ContainerStarted","Data":"f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9"}
Aug 04 21:35:06.549478 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:06.549095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr" event={"ID":"9d7f4adc-f784-474f-acd9-11cd81d4a166","Type":"ContainerStarted","Data":"835e3230d03e6ee5a3c1ace364a1fc957abe61303b394700ea918bda28187713"}
Aug 04 21:35:06.549478 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:06.549277    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:35:06.569675 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:06.569629    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr" podStartSLOduration=2.5696125739999998 podStartE2EDuration="2.569612574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:35:06.567685557 +0000 UTC m=+3281.319791546" watchObservedRunningTime="2026-08-04 21:35:06.569612574 +0000 UTC m=+3281.321718550"
Aug 04 21:35:07.455809 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:07.455773    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:35:16.550868 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:16.550839    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:35:17.557368 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:17.557323    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:35:25.992679 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:25.992646    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:35:25.999094 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:25.999071    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:35:34.102242 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.102212    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"]
Aug 04 21:35:34.107918 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.107898    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:35:34.110497 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.110465    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-80be56\""
Aug 04 21:35:34.117858 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.117834    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"]
Aug 04 21:35:34.168030 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.167995    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4j8h\" (UniqueName: \"kubernetes.io/projected/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-kube-api-access-j4j8h\") pod \"payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6\" (UID: \"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:35:34.168195 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.168067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6\" (UID: \"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:35:34.268735 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.268704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4j8h\" (UniqueName: \"kubernetes.io/projected/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-kube-api-access-j4j8h\") pod \"payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6\" (UID: \"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:35:34.268735 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.268740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6\" (UID: \"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:35:34.269425 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.269407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6\" (UID: \"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:35:34.282742 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.282712    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4j8h\" (UniqueName: \"kubernetes.io/projected/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-kube-api-access-j4j8h\") pod \"payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6\" (UID: \"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:35:34.420595 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.420522    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:35:34.761472 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:34.761449    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"]
Aug 04 21:35:34.763619 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:35:34.763592    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b03cce_a1d4_4d6d_a6d6_d26fd413cd8d.slice/crio-261ac0ae2b6308ad02c871fe6d40ec621181a2a5ab03a192d1784e75143f5c46 WatchSource:0}: Error finding container 261ac0ae2b6308ad02c871fe6d40ec621181a2a5ab03a192d1784e75143f5c46: Status 404 returned error can't find the container with id 261ac0ae2b6308ad02c871fe6d40ec621181a2a5ab03a192d1784e75143f5c46
Aug 04 21:35:35.703718 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:35.703683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6" event={"ID":"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d","Type":"ContainerStarted","Data":"33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891"}
Aug 04 21:35:35.704098 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:35.703740    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6" event={"ID":"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d","Type":"ContainerStarted","Data":"261ac0ae2b6308ad02c871fe6d40ec621181a2a5ab03a192d1784e75143f5c46"}
Aug 04 21:35:35.704098 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:35.703761    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:35:35.721691 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:35.721641    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6" podStartSLOduration=1.721626214 podStartE2EDuration="1.721626214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:35:35.720515406 +0000 UTC m=+3310.472621385" watchObservedRunningTime="2026-08-04 21:35:35.721626214 +0000 UTC m=+3310.473732190"
Aug 04 21:35:36.466772 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:36.466731    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:35:46.711125 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:35:46.711087    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:36:04.195459 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:04.195422    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fxs52_dafea131-0f06-41ca-92ff-3bc9c21e2471/payload-processing/0.log"
Aug 04 21:36:14.652444 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.652407    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"]
Aug 04 21:36:14.656148 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.656120    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.660410 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.660377    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e783f6\"/\"ipp-route-e783f6-kserve-self-signed-certs\""
Aug 04 21:36:14.660549 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.660412    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e783f6\"/\"kube-root-ca.crt\""
Aug 04 21:36:14.660549 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.660387    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e783f6\"/\"default-dockercfg-l8n8j\""
Aug 04 21:36:14.660683 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.660377    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e783f6\"/\"openshift-service-ca.crt\""
Aug 04 21:36:14.666005 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.665983    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"]
Aug 04 21:36:14.737720 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.737681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-home\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.737888 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.737735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tls-certs\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.737888 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.737762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-dshm\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.737888 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.737793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tmp-dir\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.737888 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.737821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-model-cache\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.738034 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.737899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jw6\" (UniqueName: \"kubernetes.io/projected/757cba35-1c82-4252-a7cc-b81ae10e3cb8-kube-api-access-k9jw6\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.839332 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.839293    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jw6\" (UniqueName: \"kubernetes.io/projected/757cba35-1c82-4252-a7cc-b81ae10e3cb8-kube-api-access-k9jw6\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.839563 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.839385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-home\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.839563 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.839421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tls-certs\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.839563 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.839445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-dshm\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.839563 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.839471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tmp-dir\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.839563 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.839506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-model-cache\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.839921 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.839891    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-home\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.839921 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.839902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tmp-dir\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.840072 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.839963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-model-cache\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.841767 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.841743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-dshm\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.841931 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.841913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tls-certs\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.848118 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.848075    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jw6\" (UniqueName: \"kubernetes.io/projected/757cba35-1c82-4252-a7cc-b81ae10e3cb8-kube-api-access-k9jw6\") pod \"ipp-route-e783f6-kserve-68798d9cf5-r7cs4\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") " pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:14.967050 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:14.967022    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:15.109266 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:15.109234    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"]
Aug 04 21:36:15.111227 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:36:15.111203    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod757cba35_1c82_4252_a7cc_b81ae10e3cb8.slice/crio-cae055f61e6f2a710fb23245cebb895f92469eaa68372656054321ae0565c289 WatchSource:0}: Error finding container cae055f61e6f2a710fb23245cebb895f92469eaa68372656054321ae0565c289: Status 404 returned error can't find the container with id cae055f61e6f2a710fb23245cebb895f92469eaa68372656054321ae0565c289
Aug 04 21:36:15.907433 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:15.907395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4" event={"ID":"757cba35-1c82-4252-a7cc-b81ae10e3cb8","Type":"ContainerStarted","Data":"f0d001ee29425a259022d333974069380cb2b3b7e5cc031caaf6bc75cf198286"}
Aug 04 21:36:15.907802 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:15.907447    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4" event={"ID":"757cba35-1c82-4252-a7cc-b81ae10e3cb8","Type":"ContainerStarted","Data":"cae055f61e6f2a710fb23245cebb895f92469eaa68372656054321ae0565c289"}
Aug 04 21:36:17.079691 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:17.079636    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4" podStartSLOduration=3.079618446 podStartE2EDuration="3.079618446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:36:15.927674239 +0000 UTC m=+3350.679780218" watchObservedRunningTime="2026-08-04 21:36:17.079618446 +0000 UTC m=+3351.831724450"
Aug 04 21:36:17.080487 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:17.080468    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:36:24.968091 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:24.968062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:24.968602 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:24.968377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:24.981176 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:24.981152    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:25.974639 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:25.974611    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:36:43.463088 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:43.463048    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:36:54.335996 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:54.335964    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr_9d7f4adc-f784-474f-acd9-11cd81d4a166/payload-processing/0.log"
Aug 04 21:36:54.461474 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:36:54.461445    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fxs52_dafea131-0f06-41ca-92ff-3bc9c21e2471/payload-processing/0.log"
Aug 04 21:37:01.209092 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.209054    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"]
Aug 04 21:37:01.212732 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.212715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:01.215561 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.215533    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-354c63-dockercfg-8sdjq\""
Aug 04 21:37:01.215940 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.215923    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-354c63\""
Aug 04 21:37:01.222222 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.222198    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"]
Aug 04 21:37:01.262148 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.262115    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"]
Aug 04 21:37:01.265554 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.265539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:01.279006 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.278983    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"]
Aug 04 21:37:01.289764 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.289736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfmps\" (UniqueName: \"kubernetes.io/projected/737909c6-8891-4b5a-b93c-f5f4ecf512b4-kube-api-access-nfmps\") pod \"payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb\" (UID: \"737909c6-8891-4b5a-b93c-f5f4ecf512b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:01.289920 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.289783    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737909c6-8891-4b5a-b93c-f5f4ecf512b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb\" (UID: \"737909c6-8891-4b5a-b93c-f5f4ecf512b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:01.390836 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.390804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfmps\" (UniqueName: \"kubernetes.io/projected/737909c6-8891-4b5a-b93c-f5f4ecf512b4-kube-api-access-nfmps\") pod \"payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb\" (UID: \"737909c6-8891-4b5a-b93c-f5f4ecf512b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:01.391017 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.390848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n67wr\" (UniqueName: \"kubernetes.io/projected/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-kube-api-access-n67wr\") pod \"payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp\" (UID: \"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:01.391017 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.390892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp\" (UID: \"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:01.391017 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.390918    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737909c6-8891-4b5a-b93c-f5f4ecf512b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb\" (UID: \"737909c6-8891-4b5a-b93c-f5f4ecf512b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:01.391574 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.391556    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737909c6-8891-4b5a-b93c-f5f4ecf512b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb\" (UID: \"737909c6-8891-4b5a-b93c-f5f4ecf512b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:01.399965 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.399941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfmps\" (UniqueName: \"kubernetes.io/projected/737909c6-8891-4b5a-b93c-f5f4ecf512b4-kube-api-access-nfmps\") pod \"payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb\" (UID: \"737909c6-8891-4b5a-b93c-f5f4ecf512b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:01.492052 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.492007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n67wr\" (UniqueName: \"kubernetes.io/projected/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-kube-api-access-n67wr\") pod \"payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp\" (UID: \"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:01.492052 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.492060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp\" (UID: \"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:01.492807 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.492786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp\" (UID: \"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:01.500804 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.500777    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n67wr\" (UniqueName: \"kubernetes.io/projected/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-kube-api-access-n67wr\") pod \"payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp\" (UID: \"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:01.524728 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.524701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:01.576505 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.576412    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:01.673703 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.673680    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"]
Aug 04 21:37:01.676495 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:37:01.676457    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737909c6_8891_4b5a_b93c_f5f4ecf512b4.slice/crio-5a12054def9346f1c5cbbf48f398b2c35e0f765a029ea0bc29045e39316d34b5 WatchSource:0}: Error finding container 5a12054def9346f1c5cbbf48f398b2c35e0f765a029ea0bc29045e39316d34b5: Status 404 returned error can't find the container with id 5a12054def9346f1c5cbbf48f398b2c35e0f765a029ea0bc29045e39316d34b5
Aug 04 21:37:01.732427 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:01.732404    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"]
Aug 04 21:37:01.734521 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:37:01.734492    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf47ba9_b1ab_4c8c_8c89_dde16f52b7c9.slice/crio-44c9d5ad046b697ec6361c091c46c883fc3c2890f0b7a72b31008034cc5e8ac5 WatchSource:0}: Error finding container 44c9d5ad046b697ec6361c091c46c883fc3c2890f0b7a72b31008034cc5e8ac5: Status 404 returned error can't find the container with id 44c9d5ad046b697ec6361c091c46c883fc3c2890f0b7a72b31008034cc5e8ac5
Aug 04 21:37:02.149454 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:02.149325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp" event={"ID":"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9","Type":"ContainerStarted","Data":"ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc"}
Aug 04 21:37:02.149454 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:02.149407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp" event={"ID":"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9","Type":"ContainerStarted","Data":"44c9d5ad046b697ec6361c091c46c883fc3c2890f0b7a72b31008034cc5e8ac5"}
Aug 04 21:37:02.149684 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:02.149480    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:02.151054 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:02.151023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb" event={"ID":"737909c6-8891-4b5a-b93c-f5f4ecf512b4","Type":"ContainerStarted","Data":"3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5"}
Aug 04 21:37:02.151213 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:02.151072    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb" event={"ID":"737909c6-8891-4b5a-b93c-f5f4ecf512b4","Type":"ContainerStarted","Data":"5a12054def9346f1c5cbbf48f398b2c35e0f765a029ea0bc29045e39316d34b5"}
Aug 04 21:37:02.151213 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:02.151098    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:02.168173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:02.168113    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp" podStartSLOduration=1.168098859 podStartE2EDuration="1.168098859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:37:02.167848969 +0000 UTC m=+3396.919954947" watchObservedRunningTime="2026-08-04 21:37:02.168098859 +0000 UTC m=+3396.920204835"
Aug 04 21:37:02.188255 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:02.188199    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb" podStartSLOduration=1.188178736 podStartE2EDuration="1.188178736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:37:02.186878263 +0000 UTC m=+3396.938984241" watchObservedRunningTime="2026-08-04 21:37:02.188178736 +0000 UTC m=+3396.940284713"
Aug 04 21:37:06.460030 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:06.459992    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:37:13.158206 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:13.158175    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:13.158618 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:13.158398    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:14.129842 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:14.129807    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"]
Aug 04 21:37:14.133408 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:14.133390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"
Aug 04 21:37:14.136159 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:14.136129    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 04 21:37:14.136264 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:14.136129    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-86svs\""
Aug 04 21:37:14.143272 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:14.143248    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"]
Aug 04 21:37:14.210989 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:14.210958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6914c65-fab7-42ea-aa17-e2698a971c0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk\" (UID: \"e6914c65-fab7-42ea-aa17-e2698a971c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"
Aug 04 21:37:14.315411 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:14.315375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6914c65-fab7-42ea-aa17-e2698a971c0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk\" (UID: \"e6914c65-fab7-42ea-aa17-e2698a971c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"
Aug 04 21:37:14.316005 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:14.315982    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6914c65-fab7-42ea-aa17-e2698a971c0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk\" (UID: \"e6914c65-fab7-42ea-aa17-e2698a971c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"
Aug 04 21:37:14.447066 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:14.446986    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"
Aug 04 21:37:14.792608 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:14.792583    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"]
Aug 04 21:37:14.794778 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:37:14.794750    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6914c65_fab7_42ea_aa17_e2698a971c0b.slice/crio-d2bdbd4241a0ecb3322d3176873e37dd59efa6255173fc62f938d52b379f9518 WatchSource:0}: Error finding container d2bdbd4241a0ecb3322d3176873e37dd59efa6255173fc62f938d52b379f9518: Status 404 returned error can't find the container with id d2bdbd4241a0ecb3322d3176873e37dd59efa6255173fc62f938d52b379f9518
Aug 04 21:37:15.221784 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:15.221750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk" event={"ID":"e6914c65-fab7-42ea-aa17-e2698a971c0b","Type":"ContainerStarted","Data":"3cb212fc20fc35731cac47e55474dd4156705ed470a60965034bae0cba14ce92"}
Aug 04 21:37:15.222177 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:15.221804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk" event={"ID":"e6914c65-fab7-42ea-aa17-e2698a971c0b","Type":"ContainerStarted","Data":"d2bdbd4241a0ecb3322d3176873e37dd59efa6255173fc62f938d52b379f9518"}
Aug 04 21:37:15.241599 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:15.241548    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk" podStartSLOduration=1.241534662 podStartE2EDuration="1.241534662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:37:15.240083789 +0000 UTC m=+3409.992189766" watchObservedRunningTime="2026-08-04 21:37:15.241534662 +0000 UTC m=+3409.993640641"
Aug 04 21:37:16.228181 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:16.228149    2565 generic.go:356] "Generic (PLEG): container finished" podID="e6914c65-fab7-42ea-aa17-e2698a971c0b" containerID="3cb212fc20fc35731cac47e55474dd4156705ed470a60965034bae0cba14ce92" exitCode=0
Aug 04 21:37:16.228655 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:16.228190    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk" event={"ID":"e6914c65-fab7-42ea-aa17-e2698a971c0b","Type":"ContainerDied","Data":"3cb212fc20fc35731cac47e55474dd4156705ed470a60965034bae0cba14ce92"}
Aug 04 21:37:17.371448 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:17.371427    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"
Aug 04 21:37:17.445028 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:17.444996    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6914c65-fab7-42ea-aa17-e2698a971c0b-maas-api-service-ca\") pod \"e6914c65-fab7-42ea-aa17-e2698a971c0b\" (UID: \"e6914c65-fab7-42ea-aa17-e2698a971c0b\") "
Aug 04 21:37:17.445430 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:17.445405    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6914c65-fab7-42ea-aa17-e2698a971c0b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6914c65-fab7-42ea-aa17-e2698a971c0b" (UID: "e6914c65-fab7-42ea-aa17-e2698a971c0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:37:17.546375 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:17.546268    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6914c65-fab7-42ea-aa17-e2698a971c0b-maas-api-service-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:37:18.239793 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:18.239717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk" event={"ID":"e6914c65-fab7-42ea-aa17-e2698a971c0b","Type":"ContainerDied","Data":"d2bdbd4241a0ecb3322d3176873e37dd59efa6255173fc62f938d52b379f9518"}
Aug 04 21:37:18.239793 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:18.239739    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"
Aug 04 21:37:18.239793 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:18.239764    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2bdbd4241a0ecb3322d3176873e37dd59efa6255173fc62f938d52b379f9518"
Aug 04 21:37:21.926536 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:21.926497    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"]
Aug 04 21:37:21.927658 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:21.927607    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb" podUID="737909c6-8891-4b5a-b93c-f5f4ecf512b4" containerName="payload-processing" containerID="cri-o://3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5" gracePeriod=30
Aug 04 21:37:21.931944 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:21.931916    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"]
Aug 04 21:37:21.932154 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:21.932121    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp" podUID="5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9" containerName="payload-pre-processing" containerID="cri-o://ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc" gracePeriod=30
Aug 04 21:37:22.123756 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.123711    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"]
Aug 04 21:37:22.125668 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.125644    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-354c63-bmcjk"]
Aug 04 21:37:22.215856 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.215833    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:22.218782 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.218761    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:22.262429 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.262398    2565 generic.go:356] "Generic (PLEG): container finished" podID="5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9" containerID="ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc" exitCode=0
Aug 04 21:37:22.262623 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.262449    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"
Aug 04 21:37:22.262623 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.262479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp" event={"ID":"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9","Type":"ContainerDied","Data":"ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc"}
Aug 04 21:37:22.262623 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.262534    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp" event={"ID":"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9","Type":"ContainerDied","Data":"44c9d5ad046b697ec6361c091c46c883fc3c2890f0b7a72b31008034cc5e8ac5"}
Aug 04 21:37:22.262623 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.262567    2565 scope.go:117] "RemoveContainer" containerID="ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc"
Aug 04 21:37:22.263844 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.263753    2565 generic.go:356] "Generic (PLEG): container finished" podID="737909c6-8891-4b5a-b93c-f5f4ecf512b4" containerID="3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5" exitCode=0
Aug 04 21:37:22.263844 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.263794    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"
Aug 04 21:37:22.263959 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.263847    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb" event={"ID":"737909c6-8891-4b5a-b93c-f5f4ecf512b4","Type":"ContainerDied","Data":"3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5"}
Aug 04 21:37:22.263959 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.263888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb" event={"ID":"737909c6-8891-4b5a-b93c-f5f4ecf512b4","Type":"ContainerDied","Data":"5a12054def9346f1c5cbbf48f398b2c35e0f765a029ea0bc29045e39316d34b5"}
Aug 04 21:37:22.272456 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.272428    2565 scope.go:117] "RemoveContainer" containerID="ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc"
Aug 04 21:37:22.272784 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:37:22.272765    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc\": container with ID starting with ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc not found: ID does not exist" containerID="ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc"
Aug 04 21:37:22.272846 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.272791    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc"} err="failed to get container status \"ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc\": rpc error: code = NotFound desc = could not find container \"ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc\": container with ID starting with ea9f3338aa78e5fbeb9998801c1e3effdb8855105d017b7ce2c5951092fa13bc not found: ID does not exist"
Aug 04 21:37:22.272846 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.272806    2565 scope.go:117] "RemoveContainer" containerID="3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5"
Aug 04 21:37:22.282423 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.282402    2565 scope.go:117] "RemoveContainer" containerID="3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5"
Aug 04 21:37:22.282680 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:37:22.282662    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5\": container with ID starting with 3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5 not found: ID does not exist" containerID="3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5"
Aug 04 21:37:22.282749 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.282688    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5"} err="failed to get container status \"3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5\": rpc error: code = NotFound desc = could not find container \"3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5\": container with ID starting with 3e6fe118ecc70441edc37adb85f9f6a03311452c66b0b7cfc92c974cf80c5ad5 not found: ID does not exist"
Aug 04 21:37:22.292014 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.291994    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n67wr\" (UniqueName: \"kubernetes.io/projected/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-kube-api-access-n67wr\") pod \"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9\" (UID: \"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9\") "
Aug 04 21:37:22.292106 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.292027    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737909c6-8891-4b5a-b93c-f5f4ecf512b4-plugins-config-volume\") pod \"737909c6-8891-4b5a-b93c-f5f4ecf512b4\" (UID: \"737909c6-8891-4b5a-b93c-f5f4ecf512b4\") "
Aug 04 21:37:22.292152 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.292120    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-plugins-config-volume\") pod \"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9\" (UID: \"5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9\") "
Aug 04 21:37:22.292152 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.292149    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfmps\" (UniqueName: \"kubernetes.io/projected/737909c6-8891-4b5a-b93c-f5f4ecf512b4-kube-api-access-nfmps\") pod \"737909c6-8891-4b5a-b93c-f5f4ecf512b4\" (UID: \"737909c6-8891-4b5a-b93c-f5f4ecf512b4\") "
Aug 04 21:37:22.292497 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.292472    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737909c6-8891-4b5a-b93c-f5f4ecf512b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "737909c6-8891-4b5a-b93c-f5f4ecf512b4" (UID: "737909c6-8891-4b5a-b93c-f5f4ecf512b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:37:22.292572 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.292481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9" (UID: "5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:37:22.294069 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.294044    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-kube-api-access-n67wr" (OuterVolumeSpecName: "kube-api-access-n67wr") pod "5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9" (UID: "5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9"). InnerVolumeSpecName "kube-api-access-n67wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:37:22.294165 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.294101    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737909c6-8891-4b5a-b93c-f5f4ecf512b4-kube-api-access-nfmps" (OuterVolumeSpecName: "kube-api-access-nfmps") pod "737909c6-8891-4b5a-b93c-f5f4ecf512b4" (UID: "737909c6-8891-4b5a-b93c-f5f4ecf512b4"). InnerVolumeSpecName "kube-api-access-nfmps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:37:22.392903 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.392868    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:37:22.392903 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.392897    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfmps\" (UniqueName: \"kubernetes.io/projected/737909c6-8891-4b5a-b93c-f5f4ecf512b4-kube-api-access-nfmps\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:37:22.392903 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.392906    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n67wr\" (UniqueName: \"kubernetes.io/projected/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9-kube-api-access-n67wr\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:37:22.393140 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.392915    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737909c6-8891-4b5a-b93c-f5f4ecf512b4-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:37:22.593152 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.593125    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"]
Aug 04 21:37:22.595022 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.594993    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-354c63-6d67787d5b-vwtxp"]
Aug 04 21:37:22.604403 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.604381    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"]
Aug 04 21:37:22.609452 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:22.609431    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-354c63-c98bc5b88-66qrb"]
Aug 04 21:37:23.760785 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:23.760752    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9" path="/var/lib/kubelet/pods/5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9/volumes"
Aug 04 21:37:23.761157 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:23.761109    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="737909c6-8891-4b5a-b93c-f5f4ecf512b4" path="/var/lib/kubelet/pods/737909c6-8891-4b5a-b93c-f5f4ecf512b4/volumes"
Aug 04 21:37:23.761413 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:23.761400    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6914c65-fab7-42ea-aa17-e2698a971c0b" path="/var/lib/kubelet/pods/e6914c65-fab7-42ea-aa17-e2698a971c0b/volumes"
Aug 04 21:37:24.986959 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:24.986921    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:37:33.142871 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.142836    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"]
Aug 04 21:37:33.143331 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.143241    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9" containerName="payload-pre-processing"
Aug 04 21:37:33.143331 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.143251    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9" containerName="payload-pre-processing"
Aug 04 21:37:33.143331 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.143264    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6914c65-fab7-42ea-aa17-e2698a971c0b" containerName="revoke-keys"
Aug 04 21:37:33.143331 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.143270    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6914c65-fab7-42ea-aa17-e2698a971c0b" containerName="revoke-keys"
Aug 04 21:37:33.143331 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.143290    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737909c6-8891-4b5a-b93c-f5f4ecf512b4" containerName="payload-processing"
Aug 04 21:37:33.143331 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.143295    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="737909c6-8891-4b5a-b93c-f5f4ecf512b4" containerName="payload-processing"
Aug 04 21:37:33.143601 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.143375    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="737909c6-8891-4b5a-b93c-f5f4ecf512b4" containerName="payload-processing"
Aug 04 21:37:33.143601 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.143385    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf47ba9-b1ab-4c8c-8c89-dde16f52b7c9" containerName="payload-pre-processing"
Aug 04 21:37:33.143601 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.143392    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6914c65-fab7-42ea-aa17-e2698a971c0b" containerName="revoke-keys"
Aug 04 21:37:33.148247 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.148230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"
Aug 04 21:37:33.150830 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.150806    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 04 21:37:33.150964 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.150943    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-86svs\""
Aug 04 21:37:33.154537 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.154511    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"]
Aug 04 21:37:33.303621 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.303581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b6882c-6a93-4ef6-a11b-e18a97863a2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p\" (UID: \"97b6882c-6a93-4ef6-a11b-e18a97863a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"
Aug 04 21:37:33.404821 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.404739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b6882c-6a93-4ef6-a11b-e18a97863a2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p\" (UID: \"97b6882c-6a93-4ef6-a11b-e18a97863a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"
Aug 04 21:37:33.405324 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.405303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b6882c-6a93-4ef6-a11b-e18a97863a2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p\" (UID: \"97b6882c-6a93-4ef6-a11b-e18a97863a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"
Aug 04 21:37:33.461060 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.461032    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"
Aug 04 21:37:33.802182 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:33.802155    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"]
Aug 04 21:37:33.804357 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:37:33.804316    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b6882c_6a93_4ef6_a11b_e18a97863a2f.slice/crio-a4328ba46a738893ce67c8d46bac98526e046d9f255b1ccf6e082d24e9315823 WatchSource:0}: Error finding container a4328ba46a738893ce67c8d46bac98526e046d9f255b1ccf6e082d24e9315823: Status 404 returned error can't find the container with id a4328ba46a738893ce67c8d46bac98526e046d9f255b1ccf6e082d24e9315823
Aug 04 21:37:34.330406 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:34.330366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p" event={"ID":"97b6882c-6a93-4ef6-a11b-e18a97863a2f","Type":"ContainerStarted","Data":"67349becdf495d49f53c3c4b689126c15a3860b67a644fd43e9c912b605fe698"}
Aug 04 21:37:34.330800 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:34.330432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p" event={"ID":"97b6882c-6a93-4ef6-a11b-e18a97863a2f","Type":"ContainerStarted","Data":"a4328ba46a738893ce67c8d46bac98526e046d9f255b1ccf6e082d24e9315823"}
Aug 04 21:37:34.347754 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:34.347706    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p" podStartSLOduration=1.347690555 podStartE2EDuration="1.347690555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:37:34.345828905 +0000 UTC m=+3429.097934884" watchObservedRunningTime="2026-08-04 21:37:34.347690555 +0000 UTC m=+3429.099796531"
Aug 04 21:37:35.337275 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:35.337240    2565 generic.go:356] "Generic (PLEG): container finished" podID="97b6882c-6a93-4ef6-a11b-e18a97863a2f" containerID="67349becdf495d49f53c3c4b689126c15a3860b67a644fd43e9c912b605fe698" exitCode=0
Aug 04 21:37:35.337753 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:35.337324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p" event={"ID":"97b6882c-6a93-4ef6-a11b-e18a97863a2f","Type":"ContainerDied","Data":"67349becdf495d49f53c3c4b689126c15a3860b67a644fd43e9c912b605fe698"}
Aug 04 21:37:36.485208 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:36.485183    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"
Aug 04 21:37:36.645668 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:36.639545    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b6882c-6a93-4ef6-a11b-e18a97863a2f-maas-api-service-ca\") pod \"97b6882c-6a93-4ef6-a11b-e18a97863a2f\" (UID: \"97b6882c-6a93-4ef6-a11b-e18a97863a2f\") "
Aug 04 21:37:36.645668 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:36.640411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b6882c-6a93-4ef6-a11b-e18a97863a2f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97b6882c-6a93-4ef6-a11b-e18a97863a2f" (UID: "97b6882c-6a93-4ef6-a11b-e18a97863a2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:37:36.740723 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:36.740692    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b6882c-6a93-4ef6-a11b-e18a97863a2f-maas-api-service-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:37:37.350353 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:37.350320    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"
Aug 04 21:37:37.350508 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:37.350365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p" event={"ID":"97b6882c-6a93-4ef6-a11b-e18a97863a2f","Type":"ContainerDied","Data":"a4328ba46a738893ce67c8d46bac98526e046d9f255b1ccf6e082d24e9315823"}
Aug 04 21:37:37.350508 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:37.350416    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4328ba46a738893ce67c8d46bac98526e046d9f255b1ccf6e082d24e9315823"
Aug 04 21:37:46.763775 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:46.763742    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:37:51.857566 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:51.857534    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"]
Aug 04 21:37:51.858026 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:51.857825    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr" podUID="9d7f4adc-f784-474f-acd9-11cd81d4a166" containerName="payload-processing" containerID="cri-o://f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9" gracePeriod=30
Aug 04 21:37:51.867163 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:51.867131    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"]
Aug 04 21:37:51.867390 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:51.867360    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm" podUID="909af80f-7fcd-4da0-8e07-b1546085f659" containerName="payload-pre-processing" containerID="cri-o://8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a" gracePeriod=30
Aug 04 21:37:52.013843 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.013802    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"]
Aug 04 21:37:52.016454 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.016395    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e783f6-fl29p"]
Aug 04 21:37:52.118086 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.118032    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:37:52.144836 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.144812    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:37:52.178936 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.178906    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909af80f-7fcd-4da0-8e07-b1546085f659-plugins-config-volume\") pod \"909af80f-7fcd-4da0-8e07-b1546085f659\" (UID: \"909af80f-7fcd-4da0-8e07-b1546085f659\") "
Aug 04 21:37:52.179109 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.178963    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg9bp\" (UniqueName: \"kubernetes.io/projected/9d7f4adc-f784-474f-acd9-11cd81d4a166-kube-api-access-jg9bp\") pod \"9d7f4adc-f784-474f-acd9-11cd81d4a166\" (UID: \"9d7f4adc-f784-474f-acd9-11cd81d4a166\") "
Aug 04 21:37:52.179191 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.179124    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqczd\" (UniqueName: \"kubernetes.io/projected/909af80f-7fcd-4da0-8e07-b1546085f659-kube-api-access-cqczd\") pod \"909af80f-7fcd-4da0-8e07-b1546085f659\" (UID: \"909af80f-7fcd-4da0-8e07-b1546085f659\") "
Aug 04 21:37:52.179191 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.179162    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7f4adc-f784-474f-acd9-11cd81d4a166-plugins-config-volume\") pod \"9d7f4adc-f784-474f-acd9-11cd81d4a166\" (UID: \"9d7f4adc-f784-474f-acd9-11cd81d4a166\") "
Aug 04 21:37:52.179335 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.179306    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909af80f-7fcd-4da0-8e07-b1546085f659-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "909af80f-7fcd-4da0-8e07-b1546085f659" (UID: "909af80f-7fcd-4da0-8e07-b1546085f659"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:37:52.179624 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.179601    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7f4adc-f784-474f-acd9-11cd81d4a166-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d7f4adc-f784-474f-acd9-11cd81d4a166" (UID: "9d7f4adc-f784-474f-acd9-11cd81d4a166"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:37:52.179741 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.179695    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909af80f-7fcd-4da0-8e07-b1546085f659-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:37:52.179741 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.179713    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7f4adc-f784-474f-acd9-11cd81d4a166-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:37:52.181284 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.181259    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7f4adc-f784-474f-acd9-11cd81d4a166-kube-api-access-jg9bp" (OuterVolumeSpecName: "kube-api-access-jg9bp") pod "9d7f4adc-f784-474f-acd9-11cd81d4a166" (UID: "9d7f4adc-f784-474f-acd9-11cd81d4a166"). InnerVolumeSpecName "kube-api-access-jg9bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:37:52.181457 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.181313    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909af80f-7fcd-4da0-8e07-b1546085f659-kube-api-access-cqczd" (OuterVolumeSpecName: "kube-api-access-cqczd") pod "909af80f-7fcd-4da0-8e07-b1546085f659" (UID: "909af80f-7fcd-4da0-8e07-b1546085f659"). InnerVolumeSpecName "kube-api-access-cqczd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:37:52.280481 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.280447    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg9bp\" (UniqueName: \"kubernetes.io/projected/9d7f4adc-f784-474f-acd9-11cd81d4a166-kube-api-access-jg9bp\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:37:52.280481 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.280477    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqczd\" (UniqueName: \"kubernetes.io/projected/909af80f-7fcd-4da0-8e07-b1546085f659-kube-api-access-cqczd\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:37:52.436965 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.436868    2565 generic.go:356] "Generic (PLEG): container finished" podID="9d7f4adc-f784-474f-acd9-11cd81d4a166" containerID="f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9" exitCode=0
Aug 04 21:37:52.436965 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.436918    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"
Aug 04 21:37:52.436965 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.436951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr" event={"ID":"9d7f4adc-f784-474f-acd9-11cd81d4a166","Type":"ContainerDied","Data":"f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9"}
Aug 04 21:37:52.437251 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.437008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr" event={"ID":"9d7f4adc-f784-474f-acd9-11cd81d4a166","Type":"ContainerDied","Data":"835e3230d03e6ee5a3c1ace364a1fc957abe61303b394700ea918bda28187713"}
Aug 04 21:37:52.437251 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.437043    2565 scope.go:117] "RemoveContainer" containerID="f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9"
Aug 04 21:37:52.438390 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.438363    2565 generic.go:356] "Generic (PLEG): container finished" podID="909af80f-7fcd-4da0-8e07-b1546085f659" containerID="8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a" exitCode=0
Aug 04 21:37:52.438504 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.438396    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm" event={"ID":"909af80f-7fcd-4da0-8e07-b1546085f659","Type":"ContainerDied","Data":"8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a"}
Aug 04 21:37:52.438504 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.438414    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"
Aug 04 21:37:52.438504 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.438439    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm" event={"ID":"909af80f-7fcd-4da0-8e07-b1546085f659","Type":"ContainerDied","Data":"89372ff1775d3c4b364435a8bc8ab83a52e4b17613239762b0f25b4fb5248528"}
Aug 04 21:37:52.446828 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.446815    2565 scope.go:117] "RemoveContainer" containerID="f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9"
Aug 04 21:37:52.447074 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:37:52.447054    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9\": container with ID starting with f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9 not found: ID does not exist" containerID="f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9"
Aug 04 21:37:52.447133 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.447081    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9"} err="failed to get container status \"f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9\": rpc error: code = NotFound desc = could not find container \"f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9\": container with ID starting with f6722b5d6b7cf8d6eea49e42d313c1c34f810d93cdcd5c989bfb375e2dda62f9 not found: ID does not exist"
Aug 04 21:37:52.447133 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.447097    2565 scope.go:117] "RemoveContainer" containerID="8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a"
Aug 04 21:37:52.455659 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.455642    2565 scope.go:117] "RemoveContainer" containerID="8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a"
Aug 04 21:37:52.455898 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:37:52.455880    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a\": container with ID starting with 8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a not found: ID does not exist" containerID="8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a"
Aug 04 21:37:52.455954 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.455903    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a"} err="failed to get container status \"8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a\": rpc error: code = NotFound desc = could not find container \"8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a\": container with ID starting with 8a30fbe166d76d83a73d56a262fd3726105efd088d5b0e64a57a53338cfc130a not found: ID does not exist"
Aug 04 21:37:52.462564 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.462540    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"]
Aug 04 21:37:52.464500 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.464480    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e783f6-c5b8dd968-w64bm"]
Aug 04 21:37:52.473969 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.473947    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"]
Aug 04 21:37:52.477774 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:52.477755    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e783f6-59ff9ddbc6-xc9lr"]
Aug 04 21:37:53.760800 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:53.760766    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="909af80f-7fcd-4da0-8e07-b1546085f659" path="/var/lib/kubelet/pods/909af80f-7fcd-4da0-8e07-b1546085f659/volumes"
Aug 04 21:37:53.761197 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:53.761104    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97b6882c-6a93-4ef6-a11b-e18a97863a2f" path="/var/lib/kubelet/pods/97b6882c-6a93-4ef6-a11b-e18a97863a2f/volumes"
Aug 04 21:37:53.761439 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:53.761426    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d7f4adc-f784-474f-acd9-11cd81d4a166" path="/var/lib/kubelet/pods/9d7f4adc-f784-474f-acd9-11cd81d4a166/volumes"
Aug 04 21:37:54.550740 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:54.550714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:37:57.376070 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:37:57.376035    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"]
Aug 04 21:38:12.395278 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.395217    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4" podUID="757cba35-1c82-4252-a7cc-b81ae10e3cb8" containerName="main" containerID="cri-o://f0d001ee29425a259022d333974069380cb2b3b7e5cc031caaf6bc75cf198286" gracePeriod=45
Aug 04 21:38:12.553284 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.553251    2565 generic.go:356] "Generic (PLEG): container finished" podID="757cba35-1c82-4252-a7cc-b81ae10e3cb8" containerID="f0d001ee29425a259022d333974069380cb2b3b7e5cc031caaf6bc75cf198286" exitCode=0
Aug 04 21:38:12.553464 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.553324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4" event={"ID":"757cba35-1c82-4252-a7cc-b81ae10e3cb8","Type":"ContainerDied","Data":"f0d001ee29425a259022d333974069380cb2b3b7e5cc031caaf6bc75cf198286"}
Aug 04 21:38:12.648890 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.648832    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:38:12.771082 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771051    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9jw6\" (UniqueName: \"kubernetes.io/projected/757cba35-1c82-4252-a7cc-b81ae10e3cb8-kube-api-access-k9jw6\") pod \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") "
Aug 04 21:38:12.771257 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771112    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-dshm\") pod \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") "
Aug 04 21:38:12.771257 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771136    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tmp-dir\") pod \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") "
Aug 04 21:38:12.771257 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771169    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-home\") pod \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") "
Aug 04 21:38:12.771257 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-model-cache\") pod \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") "
Aug 04 21:38:12.771488 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771282    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tls-certs\") pod \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\" (UID: \"757cba35-1c82-4252-a7cc-b81ae10e3cb8\") "
Aug 04 21:38:12.771550 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771483    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "757cba35-1c82-4252-a7cc-b81ae10e3cb8" (UID: "757cba35-1c82-4252-a7cc-b81ae10e3cb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:38:12.771550 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771530    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "757cba35-1c82-4252-a7cc-b81ae10e3cb8" (UID: "757cba35-1c82-4252-a7cc-b81ae10e3cb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:38:12.771666 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771552    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-home" (OuterVolumeSpecName: "home") pod "757cba35-1c82-4252-a7cc-b81ae10e3cb8" (UID: "757cba35-1c82-4252-a7cc-b81ae10e3cb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:38:12.771709 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771690    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:38:12.771744 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771710    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:38:12.771744 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.771724    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:38:12.773380 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.773355    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-dshm" (OuterVolumeSpecName: "dshm") pod "757cba35-1c82-4252-a7cc-b81ae10e3cb8" (UID: "757cba35-1c82-4252-a7cc-b81ae10e3cb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 21:38:12.773487 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.773437    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "757cba35-1c82-4252-a7cc-b81ae10e3cb8" (UID: "757cba35-1c82-4252-a7cc-b81ae10e3cb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 21:38:12.773487 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.773459    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757cba35-1c82-4252-a7cc-b81ae10e3cb8-kube-api-access-k9jw6" (OuterVolumeSpecName: "kube-api-access-k9jw6") pod "757cba35-1c82-4252-a7cc-b81ae10e3cb8" (UID: "757cba35-1c82-4252-a7cc-b81ae10e3cb8"). InnerVolumeSpecName "kube-api-access-k9jw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:38:12.872852 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.872824    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9jw6\" (UniqueName: \"kubernetes.io/projected/757cba35-1c82-4252-a7cc-b81ae10e3cb8-kube-api-access-k9jw6\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:38:12.872852 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.872848    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/757cba35-1c82-4252-a7cc-b81ae10e3cb8-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:38:12.872852 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:12.872858    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/757cba35-1c82-4252-a7cc-b81ae10e3cb8-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:38:13.560287 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:13.560262    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"
Aug 04 21:38:13.560724 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:13.560310    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4" event={"ID":"757cba35-1c82-4252-a7cc-b81ae10e3cb8","Type":"ContainerDied","Data":"cae055f61e6f2a710fb23245cebb895f92469eaa68372656054321ae0565c289"}
Aug 04 21:38:13.560724 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:13.560374    2565 scope.go:117] "RemoveContainer" containerID="f0d001ee29425a259022d333974069380cb2b3b7e5cc031caaf6bc75cf198286"
Aug 04 21:38:13.583264 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:13.583231    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"]
Aug 04 21:38:13.585841 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:13.585821    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e783f6/ipp-route-e783f6-kserve-68798d9cf5-r7cs4"]
Aug 04 21:38:13.761002 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:13.760969    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="757cba35-1c82-4252-a7cc-b81ae10e3cb8" path="/var/lib/kubelet/pods/757cba35-1c82-4252-a7cc-b81ae10e3cb8/volumes"
Aug 04 21:38:42.555302 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.555260    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"]
Aug 04 21:38:42.555879 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.555856    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909af80f-7fcd-4da0-8e07-b1546085f659" containerName="payload-pre-processing"
Aug 04 21:38:42.555959 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.555883    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="909af80f-7fcd-4da0-8e07-b1546085f659" containerName="payload-pre-processing"
Aug 04 21:38:42.555959 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.555930    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b6882c-6a93-4ef6-a11b-e18a97863a2f" containerName="revoke-keys"
Aug 04 21:38:42.555959 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.555940    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b6882c-6a93-4ef6-a11b-e18a97863a2f" containerName="revoke-keys"
Aug 04 21:38:42.556113 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.555964    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="757cba35-1c82-4252-a7cc-b81ae10e3cb8" containerName="main"
Aug 04 21:38:42.556113 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.555972    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="757cba35-1c82-4252-a7cc-b81ae10e3cb8" containerName="main"
Aug 04 21:38:42.556113 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.555983    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d7f4adc-f784-474f-acd9-11cd81d4a166" containerName="payload-processing"
Aug 04 21:38:42.556113 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.555992    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d7f4adc-f784-474f-acd9-11cd81d4a166" containerName="payload-processing"
Aug 04 21:38:42.556113 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.556096    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d7f4adc-f784-474f-acd9-11cd81d4a166" containerName="payload-processing"
Aug 04 21:38:42.556113 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.556107    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b6882c-6a93-4ef6-a11b-e18a97863a2f" containerName="revoke-keys"
Aug 04 21:38:42.556456 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.556130    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="757cba35-1c82-4252-a7cc-b81ae10e3cb8" containerName="main"
Aug 04 21:38:42.556456 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.556140    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="909af80f-7fcd-4da0-8e07-b1546085f659" containerName="payload-pre-processing"
Aug 04 21:38:42.559280 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.559259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"
Aug 04 21:38:42.562734 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.562710    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-86svs\""
Aug 04 21:38:42.563025 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.563010    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 04 21:38:42.575715 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.575691    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"]
Aug 04 21:38:42.657617 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.657581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45503a31-4628-4414-ac1e-7e2bbd592e0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-80be56-c4425\" (UID: \"45503a31-4628-4414-ac1e-7e2bbd592e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"
Aug 04 21:38:42.759137 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.759111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45503a31-4628-4414-ac1e-7e2bbd592e0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-80be56-c4425\" (UID: \"45503a31-4628-4414-ac1e-7e2bbd592e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"
Aug 04 21:38:42.759746 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.759727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45503a31-4628-4414-ac1e-7e2bbd592e0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-80be56-c4425\" (UID: \"45503a31-4628-4414-ac1e-7e2bbd592e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"
Aug 04 21:38:42.871226 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:42.871146    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"
Aug 04 21:38:43.018543 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:43.018512    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"]
Aug 04 21:38:43.020157 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:38:43.020128    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45503a31_4628_4414_ac1e_7e2bbd592e0f.slice/crio-ba5677a374b15ac10039dc562f60481d9fdf1d0560a0fad70de7589b7068b91a WatchSource:0}: Error finding container ba5677a374b15ac10039dc562f60481d9fdf1d0560a0fad70de7589b7068b91a: Status 404 returned error can't find the container with id ba5677a374b15ac10039dc562f60481d9fdf1d0560a0fad70de7589b7068b91a
Aug 04 21:38:43.716852 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:43.716817    2565 generic.go:356] "Generic (PLEG): container finished" podID="45503a31-4628-4414-ac1e-7e2bbd592e0f" containerID="5300a9d64fccf985e72d159b9a847ac6d82f10c352b7fc6f4db6faad2db808e5" exitCode=0
Aug 04 21:38:43.717214 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:43.716892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425" event={"ID":"45503a31-4628-4414-ac1e-7e2bbd592e0f","Type":"ContainerDied","Data":"5300a9d64fccf985e72d159b9a847ac6d82f10c352b7fc6f4db6faad2db808e5"}
Aug 04 21:38:43.717214 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:43.716945    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425" event={"ID":"45503a31-4628-4414-ac1e-7e2bbd592e0f","Type":"ContainerStarted","Data":"ba5677a374b15ac10039dc562f60481d9fdf1d0560a0fad70de7589b7068b91a"}
Aug 04 21:38:44.868841 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:44.868813    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"
Aug 04 21:38:44.983126 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:44.983039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45503a31-4628-4414-ac1e-7e2bbd592e0f-maas-api-service-ca\") pod \"45503a31-4628-4414-ac1e-7e2bbd592e0f\" (UID: \"45503a31-4628-4414-ac1e-7e2bbd592e0f\") "
Aug 04 21:38:44.983396 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:44.983374    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45503a31-4628-4414-ac1e-7e2bbd592e0f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45503a31-4628-4414-ac1e-7e2bbd592e0f" (UID: "45503a31-4628-4414-ac1e-7e2bbd592e0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:38:45.084327 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:45.084294    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45503a31-4628-4414-ac1e-7e2bbd592e0f-maas-api-service-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:38:45.736039 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:45.736011    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"
Aug 04 21:38:45.736234 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:45.736044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425" event={"ID":"45503a31-4628-4414-ac1e-7e2bbd592e0f","Type":"ContainerDied","Data":"ba5677a374b15ac10039dc562f60481d9fdf1d0560a0fad70de7589b7068b91a"}
Aug 04 21:38:45.736234 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:45.736109    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba5677a374b15ac10039dc562f60481d9fdf1d0560a0fad70de7589b7068b91a"
Aug 04 21:38:48.939734 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:48.939648    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"]
Aug 04 21:38:48.940142 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:48.939915    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6" podUID="37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d" containerName="payload-pre-processing" containerID="cri-o://33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891" gracePeriod=30
Aug 04 21:38:49.120727 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.120693    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"]
Aug 04 21:38:49.123759 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.123731    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-80be56-c4425"]
Aug 04 21:38:49.187631 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.187608    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:38:49.321897 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.321857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-plugins-config-volume\") pod \"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d\" (UID: \"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d\") "
Aug 04 21:38:49.322101 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.321974    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4j8h\" (UniqueName: \"kubernetes.io/projected/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-kube-api-access-j4j8h\") pod \"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d\" (UID: \"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d\") "
Aug 04 21:38:49.322270 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.322243    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d" (UID: "37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:38:49.324085 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.324063    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-kube-api-access-j4j8h" (OuterVolumeSpecName: "kube-api-access-j4j8h") pod "37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d" (UID: "37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d"). InnerVolumeSpecName "kube-api-access-j4j8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:38:49.423098 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.423062    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:38:49.423098 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.423092    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4j8h\" (UniqueName: \"kubernetes.io/projected/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d-kube-api-access-j4j8h\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:38:49.757974 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.757939    2565 generic.go:356] "Generic (PLEG): container finished" podID="37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d" containerID="33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891" exitCode=0
Aug 04 21:38:49.758147 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.758019    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:38:49.763368 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.762414    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45503a31-4628-4414-ac1e-7e2bbd592e0f" path="/var/lib/kubelet/pods/45503a31-4628-4414-ac1e-7e2bbd592e0f/volumes"
Aug 04 21:38:49.763368 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.762993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6" event={"ID":"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d","Type":"ContainerDied","Data":"33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891"}
Aug 04 21:38:49.763368 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.763054    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6" event={"ID":"37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d","Type":"ContainerDied","Data":"261ac0ae2b6308ad02c871fe6d40ec621181a2a5ab03a192d1784e75143f5c46"}
Aug 04 21:38:49.763368 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.763094    2565 scope.go:117] "RemoveContainer" containerID="33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891"
Aug 04 21:38:49.777652 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.777635    2565 scope.go:117] "RemoveContainer" containerID="33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891"
Aug 04 21:38:49.777911 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:38:49.777893    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891\": container with ID starting with 33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891 not found: ID does not exist" containerID="33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891"
Aug 04 21:38:49.777954 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:49.777918    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891"} err="failed to get container status \"33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891\": rpc error: code = NotFound desc = could not find container \"33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891\": container with ID starting with 33ff3c7b1be167ce63c226b6ca4235e8260709057bd949c1432dc9bbc48bb891 not found: ID does not exist"
Aug 04 21:38:51.354506 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:51.354470    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:38:57.162528 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.162493    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"]
Aug 04 21:38:57.163029 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.162928    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d" containerName="payload-pre-processing"
Aug 04 21:38:57.163029 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.162944    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d" containerName="payload-pre-processing"
Aug 04 21:38:57.163029 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.162959    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45503a31-4628-4414-ac1e-7e2bbd592e0f" containerName="revoke-keys"
Aug 04 21:38:57.163029 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.162965    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="45503a31-4628-4414-ac1e-7e2bbd592e0f" containerName="revoke-keys"
Aug 04 21:38:57.163181 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.163031    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d" containerName="payload-pre-processing"
Aug 04 21:38:57.163181 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.163045    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="45503a31-4628-4414-ac1e-7e2bbd592e0f" containerName="revoke-keys"
Aug 04 21:38:57.167568 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.167549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"
Aug 04 21:38:57.170361 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.170327    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 04 21:38:57.170474 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.170446    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-86svs\""
Aug 04 21:38:57.177254 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.177230    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"]
Aug 04 21:38:57.297357 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.297323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c86cf0-f9e4-4af7-9210-2aaa1df82b5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d\" (UID: \"70c86cf0-f9e4-4af7-9210-2aaa1df82b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"
Aug 04 21:38:57.398639 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.398599    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c86cf0-f9e4-4af7-9210-2aaa1df82b5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d\" (UID: \"70c86cf0-f9e4-4af7-9210-2aaa1df82b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"
Aug 04 21:38:57.399217 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.399196    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c86cf0-f9e4-4af7-9210-2aaa1df82b5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d\" (UID: \"70c86cf0-f9e4-4af7-9210-2aaa1df82b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"
Aug 04 21:38:57.479814 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.479787    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"
Aug 04 21:38:57.613613 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.613586    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"]
Aug 04 21:38:57.616290 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:38:57.616256    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c86cf0_f9e4_4af7_9210_2aaa1df82b5e.slice/crio-79501055d8a669088f0cedd99971e4043594425b11967f5c2be10db2e526dfb7 WatchSource:0}: Error finding container 79501055d8a669088f0cedd99971e4043594425b11967f5c2be10db2e526dfb7: Status 404 returned error can't find the container with id 79501055d8a669088f0cedd99971e4043594425b11967f5c2be10db2e526dfb7
Aug 04 21:38:57.799262 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:57.799183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d" event={"ID":"70c86cf0-f9e4-4af7-9210-2aaa1df82b5e","Type":"ContainerStarted","Data":"79501055d8a669088f0cedd99971e4043594425b11967f5c2be10db2e526dfb7"}
Aug 04 21:38:58.806218 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:58.806185    2565 generic.go:356] "Generic (PLEG): container finished" podID="70c86cf0-f9e4-4af7-9210-2aaa1df82b5e" containerID="4d8891dbf09541c26c369366b4e08c55042d381e066ea888aec70dcb21a59a58" exitCode=0
Aug 04 21:38:58.806600 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:58.806252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d" event={"ID":"70c86cf0-f9e4-4af7-9210-2aaa1df82b5e","Type":"ContainerDied","Data":"4d8891dbf09541c26c369366b4e08c55042d381e066ea888aec70dcb21a59a58"}
Aug 04 21:38:59.944144 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:38:59.944113    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"
Aug 04 21:39:00.023574 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:00.023536    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c86cf0-f9e4-4af7-9210-2aaa1df82b5e-maas-api-service-ca\") pod \"70c86cf0-f9e4-4af7-9210-2aaa1df82b5e\" (UID: \"70c86cf0-f9e4-4af7-9210-2aaa1df82b5e\") "
Aug 04 21:39:00.023907 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:00.023880    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c86cf0-f9e4-4af7-9210-2aaa1df82b5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70c86cf0-f9e4-4af7-9210-2aaa1df82b5e" (UID: "70c86cf0-f9e4-4af7-9210-2aaa1df82b5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:39:00.125190 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:00.125111    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c86cf0-f9e4-4af7-9210-2aaa1df82b5e-maas-api-service-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:39:00.817997 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:00.817960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d" event={"ID":"70c86cf0-f9e4-4af7-9210-2aaa1df82b5e","Type":"ContainerDied","Data":"79501055d8a669088f0cedd99971e4043594425b11967f5c2be10db2e526dfb7"}
Aug 04 21:39:00.817997 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:00.817993    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"
Aug 04 21:39:00.818211 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:00.818007    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79501055d8a669088f0cedd99971e4043594425b11967f5c2be10db2e526dfb7"
Aug 04 21:39:03.115483 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.115444    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"]
Aug 04 21:39:03.115962 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.115768    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx" podUID="cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4" containerName="payload-processing" containerID="cri-o://5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3" gracePeriod=30
Aug 04 21:39:03.130004 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.129967    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"]
Aug 04 21:39:03.130234 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.130208    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x" podUID="dc445817-01fa-4642-a93e-2fd34f95fcae" containerName="payload-pre-processing" containerID="cri-o://91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735" gracePeriod=30
Aug 04 21:39:03.409567 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.409530    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:39:03.412703 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.412681    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:39:03.476845 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.476813    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"]
Aug 04 21:39:03.481633 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.481609    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a189ec-jg24d"]
Aug 04 21:39:03.558319 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.558287    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd2jt\" (UniqueName: \"kubernetes.io/projected/dc445817-01fa-4642-a93e-2fd34f95fcae-kube-api-access-cd2jt\") pod \"dc445817-01fa-4642-a93e-2fd34f95fcae\" (UID: \"dc445817-01fa-4642-a93e-2fd34f95fcae\") "
Aug 04 21:39:03.558527 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.558334    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqwdn\" (UniqueName: \"kubernetes.io/projected/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-kube-api-access-wqwdn\") pod \"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4\" (UID: \"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4\") "
Aug 04 21:39:03.558527 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.558385    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc445817-01fa-4642-a93e-2fd34f95fcae-plugins-config-volume\") pod \"dc445817-01fa-4642-a93e-2fd34f95fcae\" (UID: \"dc445817-01fa-4642-a93e-2fd34f95fcae\") "
Aug 04 21:39:03.558527 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.558424    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-plugins-config-volume\") pod \"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4\" (UID: \"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4\") "
Aug 04 21:39:03.558775 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.558747    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc445817-01fa-4642-a93e-2fd34f95fcae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dc445817-01fa-4642-a93e-2fd34f95fcae" (UID: "dc445817-01fa-4642-a93e-2fd34f95fcae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:39:03.558775 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.558764    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4" (UID: "cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:39:03.560524 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.560498    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc445817-01fa-4642-a93e-2fd34f95fcae-kube-api-access-cd2jt" (OuterVolumeSpecName: "kube-api-access-cd2jt") pod "dc445817-01fa-4642-a93e-2fd34f95fcae" (UID: "dc445817-01fa-4642-a93e-2fd34f95fcae"). InnerVolumeSpecName "kube-api-access-cd2jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:39:03.560654 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.560635    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-kube-api-access-wqwdn" (OuterVolumeSpecName: "kube-api-access-wqwdn") pod "cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4" (UID: "cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4"). InnerVolumeSpecName "kube-api-access-wqwdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:39:03.660173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.660075    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:39:03.660173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.660108    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd2jt\" (UniqueName: \"kubernetes.io/projected/dc445817-01fa-4642-a93e-2fd34f95fcae-kube-api-access-cd2jt\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:39:03.660173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.660126    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqwdn\" (UniqueName: \"kubernetes.io/projected/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4-kube-api-access-wqwdn\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:39:03.660173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.660135    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc445817-01fa-4642-a93e-2fd34f95fcae-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:39:03.761099 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.761065    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c86cf0-f9e4-4af7-9210-2aaa1df82b5e" path="/var/lib/kubelet/pods/70c86cf0-f9e4-4af7-9210-2aaa1df82b5e/volumes"
Aug 04 21:39:03.840941 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.840912    2565 generic.go:356] "Generic (PLEG): container finished" podID="cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4" containerID="5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3" exitCode=0
Aug 04 21:39:03.841088 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.840958    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"
Aug 04 21:39:03.841088 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.840979    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx" event={"ID":"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4","Type":"ContainerDied","Data":"5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3"}
Aug 04 21:39:03.841088 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.841031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx" event={"ID":"cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4","Type":"ContainerDied","Data":"d21380b3a53a1240c53ee09210da2d38c54de426a9646da1e63a52af47f3adb6"}
Aug 04 21:39:03.841088 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.841060    2565 scope.go:117] "RemoveContainer" containerID="5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3"
Aug 04 21:39:03.842432 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.842369    2565 generic.go:356] "Generic (PLEG): container finished" podID="dc445817-01fa-4642-a93e-2fd34f95fcae" containerID="91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735" exitCode=0
Aug 04 21:39:03.842432 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.842424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x" event={"ID":"dc445817-01fa-4642-a93e-2fd34f95fcae","Type":"ContainerDied","Data":"91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735"}
Aug 04 21:39:03.842604 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.842463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x" event={"ID":"dc445817-01fa-4642-a93e-2fd34f95fcae","Type":"ContainerDied","Data":"524084d0d503331366d803b95657114077af3d27f633769ca84cd06ea377b32d"}
Aug 04 21:39:03.842604 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.842408    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"
Aug 04 21:39:03.850028 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.850014    2565 scope.go:117] "RemoveContainer" containerID="5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3"
Aug 04 21:39:03.850250 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:39:03.850231    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3\": container with ID starting with 5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3 not found: ID does not exist" containerID="5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3"
Aug 04 21:39:03.850294 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.850257    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3"} err="failed to get container status \"5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3\": rpc error: code = NotFound desc = could not find container \"5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3\": container with ID starting with 5edf067077233733efd0cc5e92685edbe9aa0f64ccca85a9f7202ef8719d83f3 not found: ID does not exist"
Aug 04 21:39:03.850294 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.850273    2565 scope.go:117] "RemoveContainer" containerID="91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735"
Aug 04 21:39:03.857765 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.857748    2565 scope.go:117] "RemoveContainer" containerID="91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735"
Aug 04 21:39:03.858063 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:39:03.858037    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735\": container with ID starting with 91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735 not found: ID does not exist" containerID="91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735"
Aug 04 21:39:03.858127 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.858066    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735"} err="failed to get container status \"91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735\": rpc error: code = NotFound desc = could not find container \"91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735\": container with ID starting with 91c068c448bc450957b972eb56deed047c355c5c5c513d07b9a40c20b1b5d735 not found: ID does not exist"
Aug 04 21:39:03.862163 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.862142    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"]
Aug 04 21:39:03.867334 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.867314    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a189ec-6987c6bfc4-7mdpx"]
Aug 04 21:39:03.880726 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.880704    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"]
Aug 04 21:39:03.885497 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:03.885466    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a189ec-f9f48699b-zl88x"]
Aug 04 21:39:05.352015 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:05.351981    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:39:05.760548 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:05.760520    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4" path="/var/lib/kubelet/pods/cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4/volumes"
Aug 04 21:39:05.760868 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:05.760855    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc445817-01fa-4642-a93e-2fd34f95fcae" path="/var/lib/kubelet/pods/dc445817-01fa-4642-a93e-2fd34f95fcae/volumes"
Aug 04 21:39:11.639780 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.639749    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"]
Aug 04 21:39:11.640182 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.640164    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4" containerName="payload-processing"
Aug 04 21:39:11.640226 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.640186    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4" containerName="payload-processing"
Aug 04 21:39:11.640226 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.640207    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc445817-01fa-4642-a93e-2fd34f95fcae" containerName="payload-pre-processing"
Aug 04 21:39:11.640226 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.640217    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc445817-01fa-4642-a93e-2fd34f95fcae" containerName="payload-pre-processing"
Aug 04 21:39:11.640327 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.640244    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c86cf0-f9e4-4af7-9210-2aaa1df82b5e" containerName="revoke-keys"
Aug 04 21:39:11.640327 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.640249    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c86cf0-f9e4-4af7-9210-2aaa1df82b5e" containerName="revoke-keys"
Aug 04 21:39:11.640327 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.640316    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc445817-01fa-4642-a93e-2fd34f95fcae" containerName="payload-pre-processing"
Aug 04 21:39:11.640327 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.640326    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf4c0f10-193b-4fbb-a4d2-b51a27ec82f4" containerName="payload-processing"
Aug 04 21:39:11.640476 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.640335    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c86cf0-f9e4-4af7-9210-2aaa1df82b5e" containerName="revoke-keys"
Aug 04 21:39:11.643377 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.643361    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"
Aug 04 21:39:11.647700 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.647677    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-86svs\""
Aug 04 21:39:11.647834 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.647713    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 04 21:39:11.654173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.654148    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"]
Aug 04 21:39:11.744591 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.744561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c0ab1-c54e-4f57-b383-5764f0e7cfad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg\" (UID: \"195c0ab1-c54e-4f57-b383-5764f0e7cfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"
Aug 04 21:39:11.845542 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.845511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c0ab1-c54e-4f57-b383-5764f0e7cfad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg\" (UID: \"195c0ab1-c54e-4f57-b383-5764f0e7cfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"
Aug 04 21:39:11.846082 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.846061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c0ab1-c54e-4f57-b383-5764f0e7cfad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg\" (UID: \"195c0ab1-c54e-4f57-b383-5764f0e7cfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"
Aug 04 21:39:11.953581 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:11.953505    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"
Aug 04 21:39:12.090808 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:12.090775    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"]
Aug 04 21:39:12.092628 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:39:12.092597    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195c0ab1_c54e_4f57_b383_5764f0e7cfad.slice/crio-69ae0fc61d22eef7ae070f0ad12bb98fa1ac68c46986085e65794c3bbff53416 WatchSource:0}: Error finding container 69ae0fc61d22eef7ae070f0ad12bb98fa1ac68c46986085e65794c3bbff53416: Status 404 returned error can't find the container with id 69ae0fc61d22eef7ae070f0ad12bb98fa1ac68c46986085e65794c3bbff53416
Aug 04 21:39:12.892817 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:12.892782    2565 generic.go:356] "Generic (PLEG): container finished" podID="195c0ab1-c54e-4f57-b383-5764f0e7cfad" containerID="9251c20664aee99d1711bf0f2b57702c199174ccfeddad3cf9fab9ab7c9b6578" exitCode=0
Aug 04 21:39:12.893173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:12.892825    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg" event={"ID":"195c0ab1-c54e-4f57-b383-5764f0e7cfad","Type":"ContainerDied","Data":"9251c20664aee99d1711bf0f2b57702c199174ccfeddad3cf9fab9ab7c9b6578"}
Aug 04 21:39:12.893173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:12.892878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg" event={"ID":"195c0ab1-c54e-4f57-b383-5764f0e7cfad","Type":"ContainerStarted","Data":"69ae0fc61d22eef7ae070f0ad12bb98fa1ac68c46986085e65794c3bbff53416"}
Aug 04 21:39:14.033325 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:14.033301    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"
Aug 04 21:39:14.069510 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:14.069481    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c0ab1-c54e-4f57-b383-5764f0e7cfad-maas-api-service-ca\") pod \"195c0ab1-c54e-4f57-b383-5764f0e7cfad\" (UID: \"195c0ab1-c54e-4f57-b383-5764f0e7cfad\") "
Aug 04 21:39:14.069823 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:14.069803    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195c0ab1-c54e-4f57-b383-5764f0e7cfad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "195c0ab1-c54e-4f57-b383-5764f0e7cfad" (UID: "195c0ab1-c54e-4f57-b383-5764f0e7cfad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:39:14.171295 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:14.171215    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c0ab1-c54e-4f57-b383-5764f0e7cfad-maas-api-service-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:39:14.904565 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:14.904533    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"
Aug 04 21:39:14.904744 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:14.904566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg" event={"ID":"195c0ab1-c54e-4f57-b383-5764f0e7cfad","Type":"ContainerDied","Data":"69ae0fc61d22eef7ae070f0ad12bb98fa1ac68c46986085e65794c3bbff53416"}
Aug 04 21:39:14.904744 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:14.904616    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69ae0fc61d22eef7ae070f0ad12bb98fa1ac68c46986085e65794c3bbff53416"
Aug 04 21:39:16.981594 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:16.981562    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"]
Aug 04 21:39:16.982037 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:16.981765    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5" podUID="5278aaa2-65c9-497e-bd05-5388aa4f167c" containerName="payload-processing" containerID="cri-o://09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc" gracePeriod=30
Aug 04 21:39:16.993298 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:16.993268    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"]
Aug 04 21:39:16.993563 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:16.993530    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428" podUID="548e47c8-8155-4da6-9d48-a0e340ea2ff9" containerName="payload-pre-processing" containerID="cri-o://6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d" gracePeriod=30
Aug 04 21:39:17.262904 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.262882    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:39:17.265672 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.265654    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:39:17.301623 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.301601    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvvkt\" (UniqueName: \"kubernetes.io/projected/5278aaa2-65c9-497e-bd05-5388aa4f167c-kube-api-access-rvvkt\") pod \"5278aaa2-65c9-497e-bd05-5388aa4f167c\" (UID: \"5278aaa2-65c9-497e-bd05-5388aa4f167c\") "
Aug 04 21:39:17.301748 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.301707    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frc4g\" (UniqueName: \"kubernetes.io/projected/548e47c8-8155-4da6-9d48-a0e340ea2ff9-kube-api-access-frc4g\") pod \"548e47c8-8155-4da6-9d48-a0e340ea2ff9\" (UID: \"548e47c8-8155-4da6-9d48-a0e340ea2ff9\") "
Aug 04 21:39:17.301799 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.301748    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5278aaa2-65c9-497e-bd05-5388aa4f167c-plugins-config-volume\") pod \"5278aaa2-65c9-497e-bd05-5388aa4f167c\" (UID: \"5278aaa2-65c9-497e-bd05-5388aa4f167c\") "
Aug 04 21:39:17.301799 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.301791    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/548e47c8-8155-4da6-9d48-a0e340ea2ff9-plugins-config-volume\") pod \"548e47c8-8155-4da6-9d48-a0e340ea2ff9\" (UID: \"548e47c8-8155-4da6-9d48-a0e340ea2ff9\") "
Aug 04 21:39:17.302224 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.302192    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5278aaa2-65c9-497e-bd05-5388aa4f167c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5278aaa2-65c9-497e-bd05-5388aa4f167c" (UID: "5278aaa2-65c9-497e-bd05-5388aa4f167c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:39:17.302358 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.302310    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548e47c8-8155-4da6-9d48-a0e340ea2ff9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "548e47c8-8155-4da6-9d48-a0e340ea2ff9" (UID: "548e47c8-8155-4da6-9d48-a0e340ea2ff9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 21:39:17.303806 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.303778    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5278aaa2-65c9-497e-bd05-5388aa4f167c-kube-api-access-rvvkt" (OuterVolumeSpecName: "kube-api-access-rvvkt") pod "5278aaa2-65c9-497e-bd05-5388aa4f167c" (UID: "5278aaa2-65c9-497e-bd05-5388aa4f167c"). InnerVolumeSpecName "kube-api-access-rvvkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:39:17.303891 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.303807    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548e47c8-8155-4da6-9d48-a0e340ea2ff9-kube-api-access-frc4g" (OuterVolumeSpecName: "kube-api-access-frc4g") pod "548e47c8-8155-4da6-9d48-a0e340ea2ff9" (UID: "548e47c8-8155-4da6-9d48-a0e340ea2ff9"). InnerVolumeSpecName "kube-api-access-frc4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 21:39:17.326958 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.326927    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"]
Aug 04 21:39:17.333381 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.333357    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cd644a-l9xjg"]
Aug 04 21:39:17.402705 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.402676    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frc4g\" (UniqueName: \"kubernetes.io/projected/548e47c8-8155-4da6-9d48-a0e340ea2ff9-kube-api-access-frc4g\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:39:17.402705 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.402701    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5278aaa2-65c9-497e-bd05-5388aa4f167c-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:39:17.402876 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.402711    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/548e47c8-8155-4da6-9d48-a0e340ea2ff9-plugins-config-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:39:17.402876 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.402721    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvvkt\" (UniqueName: \"kubernetes.io/projected/5278aaa2-65c9-497e-bd05-5388aa4f167c-kube-api-access-rvvkt\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\""
Aug 04 21:39:17.760443 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.760410    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195c0ab1-c54e-4f57-b383-5764f0e7cfad" path="/var/lib/kubelet/pods/195c0ab1-c54e-4f57-b383-5764f0e7cfad/volumes"
Aug 04 21:39:17.922714 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.922678    2565 generic.go:356] "Generic (PLEG): container finished" podID="5278aaa2-65c9-497e-bd05-5388aa4f167c" containerID="09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc" exitCode=0
Aug 04 21:39:17.922882 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.922718    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"
Aug 04 21:39:17.922882 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.922763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5" event={"ID":"5278aaa2-65c9-497e-bd05-5388aa4f167c","Type":"ContainerDied","Data":"09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc"}
Aug 04 21:39:17.922882 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.922825    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5" event={"ID":"5278aaa2-65c9-497e-bd05-5388aa4f167c","Type":"ContainerDied","Data":"bb61ae90f361243546fa18bc64500480394a8b3afb86b09d0ed3d8793350017c"}
Aug 04 21:39:17.922882 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.922853    2565 scope.go:117] "RemoveContainer" containerID="09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc"
Aug 04 21:39:17.924180 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.924156    2565 generic.go:356] "Generic (PLEG): container finished" podID="548e47c8-8155-4da6-9d48-a0e340ea2ff9" containerID="6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d" exitCode=0
Aug 04 21:39:17.924329 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.924184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428" event={"ID":"548e47c8-8155-4da6-9d48-a0e340ea2ff9","Type":"ContainerDied","Data":"6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d"}
Aug 04 21:39:17.924329 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.924213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428" event={"ID":"548e47c8-8155-4da6-9d48-a0e340ea2ff9","Type":"ContainerDied","Data":"44851e44dacaa7cb7cd7ee3d1802ac7d0d61951f8cab36956864dcf904ab2d50"}
Aug 04 21:39:17.924329 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.924235    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"
Aug 04 21:39:17.931651 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.931630    2565 scope.go:117] "RemoveContainer" containerID="09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc"
Aug 04 21:39:17.931923 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:39:17.931904    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc\": container with ID starting with 09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc not found: ID does not exist" containerID="09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc"
Aug 04 21:39:17.931982 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.931931    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc"} err="failed to get container status \"09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc\": rpc error: code = NotFound desc = could not find container \"09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc\": container with ID starting with 09dbf607fa5f2c90df50ddc2bed429e8b17f99f11e50c5cd20a0a1780e73c2bc not found: ID does not exist"
Aug 04 21:39:17.931982 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.931947    2565 scope.go:117] "RemoveContainer" containerID="6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d"
Aug 04 21:39:17.939571 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.939557    2565 scope.go:117] "RemoveContainer" containerID="6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d"
Aug 04 21:39:17.939821 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:39:17.939803    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d\": container with ID starting with 6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d not found: ID does not exist" containerID="6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d"
Aug 04 21:39:17.939881 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.939832    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d"} err="failed to get container status \"6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d\": rpc error: code = NotFound desc = could not find container \"6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d\": container with ID starting with 6b506ad55f557d27b31ba5a57104ed163e91b99676be9daa0980e8c13994144d not found: ID does not exist"
Aug 04 21:39:17.948979 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.948957    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"]
Aug 04 21:39:17.955118 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.955099    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cd644a-7658b54b88-nt428"]
Aug 04 21:39:17.967760 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.967735    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"]
Aug 04 21:39:17.976394 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:17.976373    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cd644a-588cc88f78-tf4j5"]
Aug 04 21:39:19.053642 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:19.053604    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pf7gc"]
Aug 04 21:39:19.760183 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:19.760149    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5278aaa2-65c9-497e-bd05-5388aa4f167c" path="/var/lib/kubelet/pods/5278aaa2-65c9-497e-bd05-5388aa4f167c/volumes"
Aug 04 21:39:19.760552 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:19.760539    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="548e47c8-8155-4da6-9d48-a0e340ea2ff9" path="/var/lib/kubelet/pods/548e47c8-8155-4da6-9d48-a0e340ea2ff9/volumes"
Aug 04 21:39:19.773173 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:19.773144    2565 pod_container_manager_linux.go:217] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d] : Timed out while waiting for systemd to remove kubepods-burstable-pod37b03cce_a1d4_4d6d_a6d6_d26fd413cd8d.slice"
Aug 04 21:39:19.773259 ip-10-0-140-103 kubenswrapper[2565]: E0804 21:39:19.773178    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d] : unable to destroy cgroup paths for cgroup [kubepods burstable pod37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d] : Timed out while waiting for systemd to remove kubepods-burstable-pod37b03cce_a1d4_4d6d_a6d6_d26fd413cd8d.slice" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6" podUID="37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d"
Aug 04 21:39:19.942690 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:19.942659    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"
Aug 04 21:39:19.960357 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:19.960318    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"]
Aug 04 21:39:19.965522 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:19.965496    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-80be56-77d4d6c55b-dvmt6"]
Aug 04 21:39:21.759973 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:21.759939    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d" path="/var/lib/kubelet/pods/37b03cce-a1d4-4d6d-a6d6-d26fd413cd8d/volumes"
Aug 04 21:39:31.218136 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:31.218104    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6889885b4f-sgccm_5ce09fc4-54bf-4662-a8a1-f73733026846/copy-manifests/0.log"
Aug 04 21:39:31.226596 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:31.226573    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6889885b4f-sgccm_5ce09fc4-54bf-4662-a8a1-f73733026846/manager/0.log"
Aug 04 21:39:31.805472 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:31.805444    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764620-55wcg_b41e0a8e-d0c7-4c0e-af6f-49eb25f2d474/cleanup/0.log"
Aug 04 21:39:33.253121 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:33.253089    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-6tcdn_d83ed8a6-f4b3-4788-8f51-546959ce78ed/kuadrant-console-plugin/0.log"
Aug 04 21:39:33.486142 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:33.486112    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-pf7gc_dcb2f898-677d-49cc-9e51-f1b07f36fe93/limitador/0.log"
Aug 04 21:39:34.238253 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:34.238223    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-586b55cbcc-tc95q_a092ad57-c542-47b9-9f78-30f22c3f6f9d/kube-auth-proxy/0.log"
Aug 04 21:39:34.719898 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:34.719872    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fxs52_dafea131-0f06-41ca-92ff-3bc9c21e2471/payload-processing/0.log"
Aug 04 21:39:35.523955 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:35.523922    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5txcw_e3a6b9be-0cf5-456d-824c-9a7b050a38bd/main/0.log"
Aug 04 21:39:40.746092 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.746061    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nzqjh/must-gather-pnh54"]
Aug 04 21:39:40.746504 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.746438    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195c0ab1-c54e-4f57-b383-5764f0e7cfad" containerName="revoke-keys"
Aug 04 21:39:40.746504 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.746450    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="195c0ab1-c54e-4f57-b383-5764f0e7cfad" containerName="revoke-keys"
Aug 04 21:39:40.746504 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.746473    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5278aaa2-65c9-497e-bd05-5388aa4f167c" containerName="payload-processing"
Aug 04 21:39:40.746504 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.746479    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5278aaa2-65c9-497e-bd05-5388aa4f167c" containerName="payload-processing"
Aug 04 21:39:40.746504 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.746489    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548e47c8-8155-4da6-9d48-a0e340ea2ff9" containerName="payload-pre-processing"
Aug 04 21:39:40.746504 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.746495    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="548e47c8-8155-4da6-9d48-a0e340ea2ff9" containerName="payload-pre-processing"
Aug 04 21:39:40.746738 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.746556    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="195c0ab1-c54e-4f57-b383-5764f0e7cfad" containerName="revoke-keys"
Aug 04 21:39:40.746738 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.746567    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5278aaa2-65c9-497e-bd05-5388aa4f167c" containerName="payload-processing"
Aug 04 21:39:40.746738 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.746575    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="548e47c8-8155-4da6-9d48-a0e340ea2ff9" containerName="payload-pre-processing"
Aug 04 21:39:40.749682 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.749650    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzqjh/must-gather-pnh54"
Aug 04 21:39:40.752545 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.752520    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nzqjh\"/\"kube-root-ca.crt\""
Aug 04 21:39:40.752679 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.752636    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nzqjh\"/\"default-dockercfg-cpcnx\""
Aug 04 21:39:40.753576 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.753558    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nzqjh\"/\"openshift-service-ca.crt\""
Aug 04 21:39:40.759280 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.759257    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzqjh/must-gather-pnh54"]
Aug 04 21:39:40.929334 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.929291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74hr\" (UniqueName: \"kubernetes.io/projected/73f0540f-41ba-4d4c-97ed-43fc35057a07-kube-api-access-s74hr\") pod \"must-gather-pnh54\" (UID: \"73f0540f-41ba-4d4c-97ed-43fc35057a07\") " pod="openshift-must-gather-nzqjh/must-gather-pnh54"
Aug 04 21:39:40.929674 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:40.929648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73f0540f-41ba-4d4c-97ed-43fc35057a07-must-gather-output\") pod \"must-gather-pnh54\" (UID: \"73f0540f-41ba-4d4c-97ed-43fc35057a07\") " pod="openshift-must-gather-nzqjh/must-gather-pnh54"
Aug 04 21:39:41.031216 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:41.031111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74hr\" (UniqueName: \"kubernetes.io/projected/73f0540f-41ba-4d4c-97ed-43fc35057a07-kube-api-access-s74hr\") pod \"must-gather-pnh54\" (UID: \"73f0540f-41ba-4d4c-97ed-43fc35057a07\") " pod="openshift-must-gather-nzqjh/must-gather-pnh54"
Aug 04 21:39:41.031216 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:41.031170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73f0540f-41ba-4d4c-97ed-43fc35057a07-must-gather-output\") pod \"must-gather-pnh54\" (UID: \"73f0540f-41ba-4d4c-97ed-43fc35057a07\") " pod="openshift-must-gather-nzqjh/must-gather-pnh54"
Aug 04 21:39:41.031548 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:41.031529    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73f0540f-41ba-4d4c-97ed-43fc35057a07-must-gather-output\") pod \"must-gather-pnh54\" (UID: \"73f0540f-41ba-4d4c-97ed-43fc35057a07\") " pod="openshift-must-gather-nzqjh/must-gather-pnh54"
Aug 04 21:39:41.039514 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:41.039489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74hr\" (UniqueName: \"kubernetes.io/projected/73f0540f-41ba-4d4c-97ed-43fc35057a07-kube-api-access-s74hr\") pod \"must-gather-pnh54\" (UID: \"73f0540f-41ba-4d4c-97ed-43fc35057a07\") " pod="openshift-must-gather-nzqjh/must-gather-pnh54"
Aug 04 21:39:41.060791 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:41.060768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzqjh/must-gather-pnh54"
Aug 04 21:39:41.200047 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:41.200020    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzqjh/must-gather-pnh54"]
Aug 04 21:39:41.202459 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:39:41.202423    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73f0540f_41ba_4d4c_97ed_43fc35057a07.slice/crio-0b6b167de68a60c88cda656e554ab0cf83dbc7dcb14c1705c47df9983b891f13 WatchSource:0}: Error finding container 0b6b167de68a60c88cda656e554ab0cf83dbc7dcb14c1705c47df9983b891f13: Status 404 returned error can't find the container with id 0b6b167de68a60c88cda656e554ab0cf83dbc7dcb14c1705c47df9983b891f13
Aug 04 21:39:42.058052 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:42.058012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzqjh/must-gather-pnh54" event={"ID":"73f0540f-41ba-4d4c-97ed-43fc35057a07","Type":"ContainerStarted","Data":"0b6b167de68a60c88cda656e554ab0cf83dbc7dcb14c1705c47df9983b891f13"}
Aug 04 21:39:43.067757 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:43.067708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzqjh/must-gather-pnh54" event={"ID":"73f0540f-41ba-4d4c-97ed-43fc35057a07","Type":"ContainerStarted","Data":"142008f64c7cf7bb542a82549fad18be5bf3dd7839d9aa4c376d22fbf6ce42a5"}
Aug 04 21:39:43.068276 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:43.067770    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzqjh/must-gather-pnh54" event={"ID":"73f0540f-41ba-4d4c-97ed-43fc35057a07","Type":"ContainerStarted","Data":"e9a9023c8eb23702b5760617e15877ab85ccad80bbfe366615a7df1f128e032f"}
Aug 04 21:39:43.089497 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:43.089434    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nzqjh/must-gather-pnh54" podStartSLOduration=2.236161836 podStartE2EDuration="3.089414444s" totalImagesPullingTime="853.252608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 21:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:39:43.087127574 +0000 UTC m=+3557.839233551" watchObservedRunningTime="2026-08-04 21:39:43.089414444 +0000 UTC m=+3557.841520422"
Aug 04 21:39:43.752580 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:43.752553    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jq4km_79262ffd-10ea-4f77-be9b-0e12ac5a1c53/global-pull-secret-syncer/0.log"
Aug 04 21:39:43.862160 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:43.862116    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qwk98_4d882eee-e214-4ea9-b271-7d1ac8f3d390/konnectivity-agent/0.log"
Aug 04 21:39:43.981588 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:43.981552    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-103.ec2.internal_8b05ec092a95b70daec77eec1e81aa44/haproxy/0.log"
Aug 04 21:39:49.069885 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:49.069850    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-6tcdn_d83ed8a6-f4b3-4788-8f51-546959ce78ed/kuadrant-console-plugin/0.log"
Aug 04 21:39:49.233428 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:49.233395    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-pf7gc_dcb2f898-677d-49cc-9e51-f1b07f36fe93/limitador/0.log"
Aug 04 21:39:50.747771 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:50.747745    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1fdccb7-12a2-4f1e-98a0-1b688b22999f/alertmanager/0.log"
Aug 04 21:39:50.772893 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:50.772866    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1fdccb7-12a2-4f1e-98a0-1b688b22999f/config-reloader/0.log"
Aug 04 21:39:50.798421 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:50.798369    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1fdccb7-12a2-4f1e-98a0-1b688b22999f/kube-rbac-proxy-web/0.log"
Aug 04 21:39:50.826539 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:50.826506    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1fdccb7-12a2-4f1e-98a0-1b688b22999f/kube-rbac-proxy/0.log"
Aug 04 21:39:50.884059 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:50.883955    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1fdccb7-12a2-4f1e-98a0-1b688b22999f/kube-rbac-proxy-metric/0.log"
Aug 04 21:39:50.915495 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:50.915461    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1fdccb7-12a2-4f1e-98a0-1b688b22999f/prom-label-proxy/0.log"
Aug 04 21:39:50.941315 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:50.941273    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1fdccb7-12a2-4f1e-98a0-1b688b22999f/init-config-reloader/0.log"
Aug 04 21:39:51.018908 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.018877    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-dkgh8_74c4817c-585c-48c7-82f9-95f4a0b1b958/kube-state-metrics/0.log"
Aug 04 21:39:51.047654 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.047623    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-dkgh8_74c4817c-585c-48c7-82f9-95f4a0b1b958/kube-rbac-proxy-main/0.log"
Aug 04 21:39:51.086489 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.086455    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-dkgh8_74c4817c-585c-48c7-82f9-95f4a0b1b958/kube-rbac-proxy-self/0.log"
Aug 04 21:39:51.157367 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.157255    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-jg6db_75b9271e-d59d-4700-a5c5-886da39c25d4/monitoring-plugin/0.log"
Aug 04 21:39:51.197940 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.197909    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pp75k_b59f4a25-b7f3-41c4-b839-2b3b2a906294/node-exporter/0.log"
Aug 04 21:39:51.220876 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.220841    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pp75k_b59f4a25-b7f3-41c4-b839-2b3b2a906294/kube-rbac-proxy/0.log"
Aug 04 21:39:51.245996 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.245963    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pp75k_b59f4a25-b7f3-41c4-b839-2b3b2a906294/init-textfile/0.log"
Aug 04 21:39:51.544731 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.544649    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b4cbfb6-1cb8-4896-9adb-1e89abf249a8/prometheus/0.log"
Aug 04 21:39:51.568443 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.568409    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b4cbfb6-1cb8-4896-9adb-1e89abf249a8/config-reloader/0.log"
Aug 04 21:39:51.594941 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.594908    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b4cbfb6-1cb8-4896-9adb-1e89abf249a8/thanos-sidecar/0.log"
Aug 04 21:39:51.618858 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.618817    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b4cbfb6-1cb8-4896-9adb-1e89abf249a8/kube-rbac-proxy-web/0.log"
Aug 04 21:39:51.655334 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.655302    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b4cbfb6-1cb8-4896-9adb-1e89abf249a8/kube-rbac-proxy/0.log"
Aug 04 21:39:51.697644 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.697545    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b4cbfb6-1cb8-4896-9adb-1e89abf249a8/kube-rbac-proxy-thanos/0.log"
Aug 04 21:39:51.721745 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.721703    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b4cbfb6-1cb8-4896-9adb-1e89abf249a8/init-config-reloader/0.log"
Aug 04 21:39:51.810426 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.810380    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-7dhc9_32c9759d-81c2-4f40-943c-eed9b9121f8b/prometheus-operator-admission-webhook/0.log"
Aug 04 21:39:51.840555 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.840474    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-649c45fdd-rjc6n_c91f55b7-fdea-4528-bcc7-3a283561bab7/telemeter-client/0.log"
Aug 04 21:39:51.867682 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.867655    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-649c45fdd-rjc6n_c91f55b7-fdea-4528-bcc7-3a283561bab7/reload/0.log"
Aug 04 21:39:51.897114 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:51.897087    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-649c45fdd-rjc6n_c91f55b7-fdea-4528-bcc7-3a283561bab7/kube-rbac-proxy/0.log"
Aug 04 21:39:53.429262 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:53.429216    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-dxf6c_adab2b9e-e2af-4560-91a9-1e996c22de08/networking-console-plugin/0.log"
Aug 04 21:39:53.967995 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:53.967959    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"]
Aug 04 21:39:53.977400 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:53.977373    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:53.979539 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:53.979509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"]
Aug 04 21:39:54.081610 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.081570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6fv\" (UniqueName: \"kubernetes.io/projected/0703e097-f41e-4faa-9cfc-dce474a675b9-kube-api-access-vf6fv\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.081812 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.081701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-lib-modules\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.081812 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.081788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-sys\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.081937 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.081827    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-proc\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.081937 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.081857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-podres\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.086419 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.086395    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/1.log"
Aug 04 21:39:54.100058 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.100028    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-2qcgg_f6321981-4380-4a31-bd58-7b64e15d9e9a/console-operator/2.log"
Aug 04 21:39:54.182923 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.182875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6fv\" (UniqueName: \"kubernetes.io/projected/0703e097-f41e-4faa-9cfc-dce474a675b9-kube-api-access-vf6fv\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.183114 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.182982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-lib-modules\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.183114 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.183054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-sys\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.183114 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.183093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-proc\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.183287 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.183121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-podres\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.183368 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.183298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-podres\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.183860 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.183832    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-lib-modules\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.183939 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.183908    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-sys\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.183996 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.183966    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0703e097-f41e-4faa-9cfc-dce474a675b9-proc\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.192633 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.192598    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6fv\" (UniqueName: \"kubernetes.io/projected/0703e097-f41e-4faa-9cfc-dce474a675b9-kube-api-access-vf6fv\") pod \"perf-node-gather-daemonset-ffprs\" (UID: \"0703e097-f41e-4faa-9cfc-dce474a675b9\") " pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.295233 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.295148    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:54.488276 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.488244    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"]
Aug 04 21:39:54.490956 ip-10-0-140-103 kubenswrapper[2565]: W0804 21:39:54.490909    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0703e097_f41e_4faa_9cfc_dce474a675b9.slice/crio-5df95dd8847604a9d01db0cdf551f1c87b64e768ff25c6641917178fba350f67 WatchSource:0}: Error finding container 5df95dd8847604a9d01db0cdf551f1c87b64e768ff25c6641917178fba350f67: Status 404 returned error can't find the container with id 5df95dd8847604a9d01db0cdf551f1c87b64e768ff25c6641917178fba350f67
Aug 04 21:39:54.736591 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:54.736559    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-qjv89_a86e1e49-9816-4c01-81a3-6e4850052b95/download-server/0.log"
Aug 04 21:39:55.165054 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:55.165008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs" event={"ID":"0703e097-f41e-4faa-9cfc-dce474a675b9","Type":"ContainerStarted","Data":"a45698b9b27d95250561c75864f039140a00bba378e7f7b630a9733c26931854"}
Aug 04 21:39:55.165239 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:55.165069    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs" event={"ID":"0703e097-f41e-4faa-9cfc-dce474a675b9","Type":"ContainerStarted","Data":"5df95dd8847604a9d01db0cdf551f1c87b64e768ff25c6641917178fba350f67"}
Aug 04 21:39:55.165239 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:55.165109    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:39:55.222748 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:55.222682    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs" podStartSLOduration=2.2226619 podStartE2EDuration="2.2226619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 21:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 21:39:55.21939279 +0000 UTC m=+3569.971498792" watchObservedRunningTime="2026-08-04 21:39:55.2226619 +0000 UTC m=+3569.974767879"
Aug 04 21:39:56.604408 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:56.604380    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zr7n9_d6f2150f-4332-4f28-abe4-72de91210bfa/dns/0.log"
Aug 04 21:39:56.635058 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:56.635028    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zr7n9_d6f2150f-4332-4f28-abe4-72de91210bfa/kube-rbac-proxy/0.log"
Aug 04 21:39:56.721283 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:56.721255    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qn2gs_5302fdd4-a446-4010-8115-a4ac2915fcc6/dns-node-resolver/0.log"
Aug 04 21:39:57.472997 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:57.472966    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xltcv_430b112c-c5d1-4736-916b-6577cdc6464a/node-ca/0.log"
Aug 04 21:39:58.762105 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:58.762073    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-586b55cbcc-tc95q_a092ad57-c542-47b9-9f78-30f22c3f6f9d/kube-auth-proxy/0.log"
Aug 04 21:39:58.902526 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:58.902487    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fxs52_dafea131-0f06-41ca-92ff-3bc9c21e2471/payload-processing/0.log"
Aug 04 21:39:59.726852 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:39:59.726814    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4nrkd_cbe77e45-7755-4781-93b4-75fe732d1507/serve-healthcheck-canary/0.log"
Aug 04 21:40:00.428769 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:00.428726    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2rgn_13da4fd0-9ad4-4721-88fa-7555047c7b69/kube-rbac-proxy/0.log"
Aug 04 21:40:00.452812 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:00.452782    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2rgn_13da4fd0-9ad4-4721-88fa-7555047c7b69/exporter/0.log"
Aug 04 21:40:00.478694 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:00.478665    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2rgn_13da4fd0-9ad4-4721-88fa-7555047c7b69/extractor/0.log"
Aug 04 21:40:01.182435 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:01.182409    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nzqjh/perf-node-gather-daemonset-ffprs"
Aug 04 21:40:03.496251 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:03.496221    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6889885b4f-sgccm_5ce09fc4-54bf-4662-a8a1-f73733026846/manager/0.log"
Aug 04 21:40:03.521710 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:03.521679    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6889885b4f-sgccm_5ce09fc4-54bf-4662-a8a1-f73733026846/copy-manifests/0.log"
Aug 04 21:40:04.824176 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:04.824121    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7564c45d7d-r2jps_4108f0f3-7453-4ff5-b8c6-c6fc838c355e/manager/0.log"
Aug 04 21:40:04.883974 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:04.883949    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-586b55cbcc-tc95q_a092ad57-c542-47b9-9f78-30f22c3f6f9d/kube-auth-proxy/0.log"
Aug 04 21:40:04.987857 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:04.987822    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fxs52_dafea131-0f06-41ca-92ff-3bc9c21e2471/payload-processing/0.log"
Aug 04 21:40:06.774553 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:06.774501    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-586b55cbcc-tc95q_a092ad57-c542-47b9-9f78-30f22c3f6f9d/kube-auth-proxy/0.log"
Aug 04 21:40:06.882398 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:06.882369    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fxs52_dafea131-0f06-41ca-92ff-3bc9c21e2471/payload-processing/0.log"
Aug 04 21:40:10.445586 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:10.445559    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-fdk5z_cb97b82b-f635-4e6d-8c35-354c0e67ef49/kube-storage-version-migrator-operator/1.log"
Aug 04 21:40:10.446584 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:10.446567    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-fdk5z_cb97b82b-f635-4e6d-8c35-354c0e67ef49/kube-storage-version-migrator-operator/0.log"
Aug 04 21:40:11.573546 ip-10-0-140-103 kubenswrapper[2565]: I0804 21:40:11.573510    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4kb2h_f07b810a-cc3a-40b5-affa-6aa75dbd263a/kube-multus/0.log"