Sep 16 21:22:40.499674 ip-10-0-140-93 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 21:22:40.885740 ip-10-0-140-93 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 21:22:40.885740 ip-10-0-140-93 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 21:22:40.885740 ip-10-0-140-93 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 21:22:40.885740 ip-10-0-140-93 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 21:22:40.885740 ip-10-0-140-93 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 21:22:40.886422 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.885797    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 21:22:40.888631 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888617    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:40.888631 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888631    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888635    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888639    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888642    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888645    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888648    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888651    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888654    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888656    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888659    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888661    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888664    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888667    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888670    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888672    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888676    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888680    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888683    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888686    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888689    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:40.888694 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888698    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888701    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888704    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888707    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888710    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888713    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888716    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888718    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888721    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888723    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888727    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888730    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888733    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888736    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888739    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888741    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888744    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888746    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888749    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888751    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:40.889170 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888754    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888757    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888759    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888762    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888765    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888767    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888770    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888772    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888775    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888777    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888780    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888782    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888785    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888788    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888790    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888793    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888795    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888798    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888800    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:40.889690 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888803    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888805    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888808    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888811    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888813    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888816    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888819    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888821    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888824    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888826    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888829    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888832    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888834    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888837    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888840    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888843    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888845    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888848    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888850    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888853    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:40.890135 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888855    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888858    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888861    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888864    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888866    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888870    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888872    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888875    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888878    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888880    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888884    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888888    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888891    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888893    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888896    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888899    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888902    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888905    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.888907    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889602    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:40.890617 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889609    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889612    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889615    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889618    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889627    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889629    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889633    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889636    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889639    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889642    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889644    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889647    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889650    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889652    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889655    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889657    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889660    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889663    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889666    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889668    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:40.891121 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889671    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889673    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889676    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889679    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889681    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889684    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889687    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889689    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889694    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889700    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889703    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889706    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889708    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889711    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889714    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889716    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889719    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889722    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889725    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:40.891634 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889728    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889730    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889733    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889735    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889738    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889741    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889744    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889746    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889749    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889751    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889754    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889757    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889760    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889762    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889765    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889767    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889770    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889773    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889776    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889778    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889781    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:40.892096 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889783    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889786    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889789    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889791    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889793    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889797    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889801    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889805    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889807    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889810    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889814    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889817    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889820    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889823    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889827    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889831    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889834    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889837    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889840    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:40.892727 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889842    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889845    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889849    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889852    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889854    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889857    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889859    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889862    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889865    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889867    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889870    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889872    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889875    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889878    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889880    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889883    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889886    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889888    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.889891    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:40.893194 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890313    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890318    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890322    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890327    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890339    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890342    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890345    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890348    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890351    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890354    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890356    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890359    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890362    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890364    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890367    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890370    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890373    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890375    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890378    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:40.893701 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890381    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890383    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890386    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890389    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890391    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890407    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890409    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890412    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890415    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890418    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890421    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890423    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890426    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890428    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890431    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890433    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890436    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890439    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890447    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890450    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:40.894187 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890453    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890455    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890458    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890460    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890463    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890465    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890468    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890470    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890473    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890475    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890477    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890480    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890482    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890485    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890487    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890490    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890492    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890494    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890497    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890500    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:40.894679 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890502    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890505    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890507    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890510    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890513    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890515    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890519    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890521    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890524    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890527    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890531    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890540    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890543    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890547    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890550    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890553    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890556    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890559    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890562    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:40.895168 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890564    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890567    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890569    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890572    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890574    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890577    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890580    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890582    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890584    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890587    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890589    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890592    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890594    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890597    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890599    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890610    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890613    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890616    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890618    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890621    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:40.895702 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.890625    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890699    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890743    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890750    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890754    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890765    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890768    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890772    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890776    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890779    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890782    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890786    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890789    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890792    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890795    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890798    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890801    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890803    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890806    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890811    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890814    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890817    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890820    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890823    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 21:22:40.896183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890827    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890830    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890833    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890837    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890839    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890842    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890845    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890848    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890853    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890857    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890860    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890863    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890865    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890868    2567 flags.go:64] FLAG: --enable-server="true"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890872    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890876    2567 flags.go:64] FLAG: --event-burst="100"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890879    2567 flags.go:64] FLAG: --event-qps="50"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890882    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890885    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890888    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890892    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890895    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890897    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890900    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890903    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 21:22:40.896772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890906    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890909    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890912    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890914    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890917    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890920    2567 flags.go:64] FLAG: --feature-gates=""
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890924    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890926    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890929    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890932    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890935    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890938    2567 flags.go:64] FLAG: --help="false"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890941    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-140-93.ec2.internal"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890944    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890947    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890954    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890958    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890962    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890964    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890967    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890970    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890973    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890976    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890979    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 21:22:40.897468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890982    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890985    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890988    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890991    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890994    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.890997    2567 flags.go:64] FLAG: --lock-file=""
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891000    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891002    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891005    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891010    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891013    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891016    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891018    2567 flags.go:64] FLAG: --logging-format="text"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891021    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891024    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891028    2567 flags.go:64] FLAG: --manifest-url=""
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891030    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891034    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891038    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891042    2567 flags.go:64] FLAG: --max-pods="110"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891045    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891048    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891051    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891055    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891058    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 21:22:40.898083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891061    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891063    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891070    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891073    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891076    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891079    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891082    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891088    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891091    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891094    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891097    2567 flags.go:64] FLAG: --port="10250"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891100    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891103    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04d430670eb46f726"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891106    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891108    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891111    2567 flags.go:64] FLAG: --register-node="true"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891114    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891122    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891125    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891128    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891131    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891134    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891138    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891141    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891144    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 21:22:40.898707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891147    2567 flags.go:64] FLAG: --runonce="false"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891150    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891152    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891155    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891158    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891162    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891165    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891168    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891171    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891174    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891177    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891179    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891182    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891185    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891189    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891191    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891197    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891200    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891203    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891207    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891210    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891212    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891215    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891218    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891221    2567 flags.go:64] FLAG: --v="2"
Sep 16 21:22:40.899320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891225    2567 flags.go:64] FLAG: --version="false"
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891228    2567 flags.go:64] FLAG: --vmodule=""
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891232    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.891235    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891363    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891366    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891372    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891375    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891378    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891381    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891383    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891386    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891390    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891405    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891408    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891411    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891413    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891416    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891418    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891421    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:40.899966 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891424    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891426    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891430    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891432    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891435    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891438    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891440    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891443    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891445    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891448    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891450    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891453    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891456    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891458    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891461    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891463    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891466    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891469    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891472    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:40.900492 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891475    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891478    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891481    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891483    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891486    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891490    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891492    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891495    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891498    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891500    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891503    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891505    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891508    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891511    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891513    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891516    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891519    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891522    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891524    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891529    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:40.900957 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891532    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891535    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891538    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891541    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891543    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891546    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891548    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891551    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891553    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891556    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891559    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891564    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891567    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891570    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891572    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891575    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891578    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891582    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891585    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891587    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:40.901478 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891590    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891593    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891595    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891598    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891600    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891603    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891605    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891608    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891612    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891616    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891618    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891621    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891623    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891626    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891629    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891631    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891634    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891636    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891639    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:40.901976 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891642    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891645    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891647    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891650    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.891654    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.892257    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.898994    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.899008    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.900305    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.900313    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900369    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900374    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900377    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900380    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900383    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:40.902459 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900386    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900390    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900404    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900407    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900410    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900413    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900415    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900419    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900421    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900424    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900427    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900430    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900433    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900437    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900440    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900442    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900445    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900448    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900450    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900453    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:40.902839 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900455    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900458    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900461    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900464    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900468    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900470    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900473    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900476    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900480    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900484    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900487    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900491    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900494    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900498    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900501    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900504    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900507    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900510    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:40.903318 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900513    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900516    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900519    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900522    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900525    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900528    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900531    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900534    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900537    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900539    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900542    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900545    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900547    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900550    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900553    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900555    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900558    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900561    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900564    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900567    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:40.903765 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900569    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900572    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900575    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900577    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900580    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900583    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900585    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900588    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900591    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900593    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900596    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900598    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900601    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900603    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900606    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900609    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900611    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900613    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900616    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900619    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:40.904297 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900622    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900624    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900628    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900630    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900633    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900635    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900638    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900640    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900643    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900646    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900649    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900651    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900654    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900657    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900659    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:40.904798 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900662    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.900666    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900778    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900783    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900787    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900790    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900793    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900795    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900798    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900800    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900804    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900806    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900809    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900815    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900818    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:40.905160 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900821    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900823    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900825    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900828    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900831    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900834    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900837    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900839    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900842    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900845    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900847    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900849    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900852    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900855    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900857    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900860    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900862    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900865    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900867    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900870    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:40.905637 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900872    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900875    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900878    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900880    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900883    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900885    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900888    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900890    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900892    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900895    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900898    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900901    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900904    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900906    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900909    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900911    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900914    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900916    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900920    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:40.906123 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900923    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900927    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900929    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900932    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900935    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900938    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900941    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900944    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900947    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900949    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900953    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900957    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900960    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900963    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900966    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900969    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900972    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900975    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900977    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:40.906605 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900980    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900982    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900985    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900987    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900990    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900992    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900995    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.900997    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901000    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901002    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901005    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901007    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901010    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901012    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901015    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901017    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901020    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901023    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901025    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901028    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:40.907065 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901031    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901033    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901036    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901039    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901041    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901044    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901046    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:40.901049    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.901054    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.901146    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.903811    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.904760    2567 server.go:1019] "Starting client certificate rotation"
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.904863    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 21:22:40.907553 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.904898    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 21:22:40.909750 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.909735    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 21:22:40.911041 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.911025    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 21:22:40.920174 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.920158    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 16 21:22:40.923748 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.923732    2567 log.go:25] "Validated CRI v1 image API"
Sep 16 21:22:40.925944 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.925932    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 21:22:40.930075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.930056    2567 fs.go:135] Filesystem UUIDs: map[512f0746-e167-45d5-a3f7-eced3e3b20e8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 aef025f9-9bae-40fa-8899-27cf47f08584:/dev/nvme0n1p3]
Sep 16 21:22:40.930150 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.930073    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 21:22:40.936251 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.936143    2567 manager.go:216] Machine: {Timestamp:2026-09-16 21:22:40.934279812 +0000 UTC m=+0.319962914 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098218 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28adaf3d06e3541f81676c76fc6f50 SystemUUID:ec28adaf-3d06-e354-1f81-676c76fc6f50 BootID:297299aa-58af-48d2-b12e-2094749e24fa Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:02:e0:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:02:e0:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:1e:72:76:24:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 21:22:40.936251 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.936246    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 21:22:40.936351 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.936320    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 21:22:40.939405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.939367    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 21:22:40.939573 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.939412    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-93.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 21:22:40.940200 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.940192    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 21:22:40.940229 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.940204    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 21:22:40.940229 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.940212    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 21:22:40.940455 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.940439    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 21:22:40.940990 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.940978    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 21:22:40.941779 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.941769    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 21:22:40.945448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.945437    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 21:22:40.945562 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.945553    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 21:22:40.945597 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.945580    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 21:22:40.945597 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.945595    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 21:22:40.945652 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.945612    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 21:22:40.945652 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.945621    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 16 21:22:40.945652 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.945629    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 21:22:40.946060 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.946049    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 21:22:40.946060 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.946067    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 21:22:40.948388 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.948374    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 21:22:40.950103 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.950089    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 21:22:40.950151 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.950119    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 21:22:40.952600 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952586    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 21:22:40.952673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952605    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 21:22:40.952673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952612    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 21:22:40.952673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952617    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 21:22:40.952673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952626    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 21:22:40.952673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952631    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 21:22:40.952673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952644    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 21:22:40.952673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952649    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 21:22:40.952673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952659    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 21:22:40.952673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952664    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 21:22:40.952673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952674    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 21:22:40.952673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.952679    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 21:22:40.953469 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.953458    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 21:22:40.953504 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.953470    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 21:22:40.955640 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.955624    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-93.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 21:22:40.955821 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:40.955800    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 21:22:40.955821 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:40.955803    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 21:22:40.957155 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.957141    2567 server.go:1268] "Started kubelet"
Sep 16 21:22:40.957234 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.957204    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 21:22:40.957801 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.957751    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 21:22:40.957882 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.957820    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 21:22:40.958447 ip-10-0-140-93 systemd[1]: Started Kubernetes Kubelet.
Sep 16 21:22:40.962961 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.962932    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 21:22:40.964197 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.964184    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 21:22:40.964390 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:40.964333    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 21:22:40.964557 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.964530    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56r9t"
Sep 16 21:22:40.967433 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.967416    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 21:22:40.967433 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.967430    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 21:22:40.967607 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:40.967584    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:40.967931 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.967913    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56r9t"
Sep 16 21:22:40.968056 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968017    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 21:22:40.968120 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968061    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 21:22:40.968205 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968191    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 21:22:40.968264 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968206    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 21:22:40.968408 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968377    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 21:22:40.968600 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968586    2567 factory.go:153] Registering CRI-O factory
Sep 16 21:22:40.968697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968603    2567 factory.go:223] Registration of the crio container factory successfully
Sep 16 21:22:40.968697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968649    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 21:22:40.968697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968659    2567 factory.go:55] Registering systemd factory
Sep 16 21:22:40.968697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968667    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 16 21:22:40.968697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968681    2567 factory.go:103] Registering Raw factory
Sep 16 21:22:40.968924 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.968707    2567 manager.go:1196] Started watching for new ooms in manager
Sep 16 21:22:40.969089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.969077    2567 manager.go:349] Starting recovery of all containers
Sep 16 21:22:40.971243 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:40.971208    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 21:22:40.971327 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:40.971308    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 21:22:40.972321 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:40.971353    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-93.ec2.internal.18d5ea0b3e1ab1ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-93.ec2.internal,UID:ip-10-0-140-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-93.ec2.internal,},FirstTimestamp:2026-09-16 21:22:40.957116844 +0000 UTC m=+0.342799919,LastTimestamp:2026-09-16 21:22:40.957116844 +0000 UTC m=+0.342799919,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-93.ec2.internal,}"
Sep 16 21:22:40.980931 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.980780    2567 manager.go:354] Recovery completed
Sep 16 21:22:40.986574 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.986560    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:40.988690 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.988673    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:40.988758 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.988705    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:40.988758 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.988716    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:40.989337 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.989320    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 21:22:40.989337 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.989334    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 21:22:40.989484 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.989341    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:40.989484 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.989351    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 21:22:40.990080 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.990062    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:40.990172 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.990089    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:40.990172 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.990101    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:40.990639 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.990627    2567 policy_none.go:49] "None policy: Start"
Sep 16 21:22:40.990685 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.990644    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 21:22:40.990685 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.990653    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 21:22:40.991724 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:40.991712    2567 policy_none.go:47] "Start"
Sep 16 21:22:41.016969 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.016953    2567 manager.go:337] "Starting Device Plugin manager"
Sep 16 21:22:41.017051 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.016994    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 21:22:41.017051 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.017008    2567 server.go:87] "Starting device plugin registration server"
Sep 16 21:22:41.017375 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.017360    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 21:22:41.017487 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.017375    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 21:22:41.017720 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.017599    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 21:22:41.017720 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.017671    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 21:22:41.017720 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.017680    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 21:22:41.018623 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.018221    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 21:22:41.018623 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.018273    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:41.100231 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.100209    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 21:22:41.101449 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.101429    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 21:22:41.101541 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.101485    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 21:22:41.101541 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.101511    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 21:22:41.101653 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.101591    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 21:22:41.103970 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.103947    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:41.118043 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.118024    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:41.120303 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.120289    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:41.120374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.120320    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:41.120374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.120331    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:41.120374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.120351    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.130032 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.130020    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.130092 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.130039    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-93.ec2.internal\": node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:41.147302 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.147250    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:41.202639 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.202616    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-93.ec2.internal"]
Sep 16 21:22:41.202712 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.202684    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:41.203630 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.203616    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:41.203688 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.203640    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:41.203688 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.203649    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:41.204803 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.204790    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:41.204928 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.204915    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.204965 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.204944    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:41.205501 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.205485    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:41.205501 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.205498    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:41.205610 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.205509    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:41.205610 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.205522    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:41.205610 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.205528    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:41.205610 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.205532    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:41.206803 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.206790    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.206845 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.206824    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:41.207945 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.207931    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:41.208021 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.207951    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:41.208021 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.207965    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:41.232015 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.231994    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-93.ec2.internal\" not found" node="ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.236599 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.236568    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-93.ec2.internal\" not found" node="ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.248345 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.248332    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:41.270164 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.270145    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6bab6030418b48a95c5e7134d86b1cea-config\") pod \"kube-apiserver-proxy-ip-10-0-140-93.ec2.internal\" (UID: \"6bab6030418b48a95c5e7134d86b1cea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.270220 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.270166    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aef40cca25998ea7c58dc54dacd5bd30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal\" (UID: \"aef40cca25998ea7c58dc54dacd5bd30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.270220 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.270182    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aef40cca25998ea7c58dc54dacd5bd30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal\" (UID: \"aef40cca25998ea7c58dc54dacd5bd30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.349140 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.349120    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:41.370462 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.370434    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6bab6030418b48a95c5e7134d86b1cea-config\") pod \"kube-apiserver-proxy-ip-10-0-140-93.ec2.internal\" (UID: \"6bab6030418b48a95c5e7134d86b1cea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.370527 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.370466    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aef40cca25998ea7c58dc54dacd5bd30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal\" (UID: \"aef40cca25998ea7c58dc54dacd5bd30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.370527 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.370484    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aef40cca25998ea7c58dc54dacd5bd30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal\" (UID: \"aef40cca25998ea7c58dc54dacd5bd30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.370597 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.370537    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6bab6030418b48a95c5e7134d86b1cea-config\") pod \"kube-apiserver-proxy-ip-10-0-140-93.ec2.internal\" (UID: \"6bab6030418b48a95c5e7134d86b1cea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.370597 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.370546    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aef40cca25998ea7c58dc54dacd5bd30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal\" (UID: \"aef40cca25998ea7c58dc54dacd5bd30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.370597 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.370546    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aef40cca25998ea7c58dc54dacd5bd30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal\" (UID: \"aef40cca25998ea7c58dc54dacd5bd30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.449848 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.449828    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:41.534383 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.534355    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.540234 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.540215    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:41.550255 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.550231    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:41.651752 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.651724    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:41.753885 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.753830    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:41.854388 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.854366    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:41.854488 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.854455    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:41.904420 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.904386    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 21:22:41.904807 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.904503    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 21:22:41.904807 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.904551    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 21:22:41.955004 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:41.954975    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-93.ec2.internal\" not found"
Sep 16 21:22:41.972571 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.972537    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 21:17:41 +0000 UTC" deadline="2028-05-25 19:08:35.67073734 +0000 UTC"
Sep 16 21:22:41.972571 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:41.972565    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14805h45m53.698175754s"
Sep 16 21:22:42.030410 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.030323    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:42.067743 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.067722    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:42.078918 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.078901    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:22:42.080889 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.080863    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 21:22:42.081817 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.081800    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-93.ec2.internal"
Sep 16 21:22:42.087856 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.087840    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 21:22:42.100326 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:42.100298    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bab6030418b48a95c5e7134d86b1cea.slice/crio-21f18149e85ec7bf1a119ee1ca8bbdfd4fa3de3ccef2009c742f434986b2483a WatchSource:0}: Error finding container 21f18149e85ec7bf1a119ee1ca8bbdfd4fa3de3ccef2009c742f434986b2483a: Status 404 returned error can't find the container with id 21f18149e85ec7bf1a119ee1ca8bbdfd4fa3de3ccef2009c742f434986b2483a
Sep 16 21:22:42.104898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.104861    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-93.ec2.internal" event={"ID":"6bab6030418b48a95c5e7134d86b1cea","Type":"ContainerStarted","Data":"21f18149e85ec7bf1a119ee1ca8bbdfd4fa3de3ccef2009c742f434986b2483a"}
Sep 16 21:22:42.106043 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.106020    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal" event={"ID":"aef40cca25998ea7c58dc54dacd5bd30","Type":"ContainerStarted","Data":"5cf4684368fa08b1d749a17e3d92c8a4ef06ac5473e9da5abffe97b75a753976"}
Sep 16 21:22:42.154343 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.154320    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:42.947894 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.947867    2567 apiserver.go:52] "Watching apiserver"
Sep 16 21:22:42.960065 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.959042    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 21:22:42.961389 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.960901    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-pgpsw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal","openshift-multus/multus-additional-cni-plugins-s86bb","openshift-multus/multus-sdq4k","openshift-multus/network-metrics-daemon-hn2j5","openshift-network-diagnostics/network-check-target-6wzrh","openshift-network-operator/iptables-alerter-7jhdj","kube-system/global-pull-secret-syncer-j92fr","kube-system/kube-apiserver-proxy-ip-10-0-140-93.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7","openshift-cluster-node-tuning-operator/tuned-swwmb","openshift-dns/node-resolver-j4dhm","openshift-ovn-kubernetes/ovnkube-node-82sp7","kube-system/konnectivity-agent-khthp"]
Sep 16 21:22:42.963017 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.962995    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7jhdj"
Sep 16 21:22:42.964490 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.964424    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.965381 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.965356    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:22:42.965512 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.965465    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 21:22:42.965585 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.965465    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 21:22:42.965646 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.965584    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9cc4r\""
Sep 16 21:22:42.965896 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.965874    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:42.966327 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.966302    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:22:42.966752 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.966729    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 21:22:42.966897 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.966877    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n5rx9\""
Sep 16 21:22:42.967138 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.967121    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.967735 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.967714    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 21:22:42.968545 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.968523    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 21:22:42.968797 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.968777    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9fflp\""
Sep 16 21:22:42.969063 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.969041    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 21:22:42.969884 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.969146    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 21:22:42.969884 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.969366    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 21:22:42.969884 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.969411    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 21:22:42.969884 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.969543    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 21:22:42.969884 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.969589    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-22xjm\""
Sep 16 21:22:42.969884 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.969758    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:42.969884 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.969767    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:42.969884 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:42.969852    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:22:42.970330 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:42.969983    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:22:42.971249 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.971229    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:22:42.972834 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.972790    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:42.972932 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:42.972857    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:22:42.973572 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.973553    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lx29h\""
Sep 16 21:22:42.973820 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.973801    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 21:22:42.973911 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.973896    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 21:22:42.974669 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.974394    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j4dhm"
Sep 16 21:22:42.976083 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.976061    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:42.976794 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.976774    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n2nfl\""
Sep 16 21:22:42.976913 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.976895    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 21:22:42.977032 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.976989    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 21:22:42.977501 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.977481    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pgpsw"
Sep 16 21:22:42.978546 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.978524    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 21:22:42.978685 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.978534    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 21:22:42.978775 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.978738    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sdprx\""
Sep 16 21:22:42.978775 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.978741    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 21:22:42.978899 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.978881    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 21:22:42.979193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.979171    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:42.979461 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.979439    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:42.979618 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.979599    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-sysctl-conf\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.979751 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.979734    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-systemd\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.979913 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.979896    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-cnibin\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.980057 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980041    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be456967-5d82-446a-be28-ded3400718b6-multus-daemon-config\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.980167 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/434cf180-e16d-43be-9860-18996cf3c1b2-system-cni-dir\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:42.980247 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9m4\" (UniqueName: \"kubernetes.io/projected/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-kube-api-access-4b9m4\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:42.980325 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980255    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-sys\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.980325 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980280    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-multus-cni-dir\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.980451 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980305    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-etc-kubernetes\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.980451 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980348    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvxn9\" (UniqueName: \"kubernetes.io/projected/be456967-5d82-446a-be28-ded3400718b6-kube-api-access-pvxn9\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.980451 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-multus-socket-dir-parent\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.980451 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-run-k8s-cni-cncf-io\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.980451 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980429    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-run-netns\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.980710 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980453    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-run-multus-certs\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.980710 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980504    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72tfb\" (UniqueName: \"kubernetes.io/projected/72686aad-8df7-4b7c-b650-e169bc50ac5a-kube-api-access-72tfb\") pod \"iptables-alerter-7jhdj\" (UID: \"72686aad-8df7-4b7c-b650-e169bc50ac5a\") " pod="openshift-network-operator/iptables-alerter-7jhdj"
Sep 16 21:22:42.980710 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980555    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-var-lib-kubelet\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.980710 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980577    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-run\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.980710 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980603    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-lib-modules\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.980710 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980626    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-host\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.980710 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980651    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16268f79-7553-4246-a98a-c086cc926e07-tmp\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.980710 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980675    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spjfw\" (UniqueName: \"kubernetes.io/projected/16268f79-7553-4246-a98a-c086cc926e07-kube-api-access-spjfw\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.981114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980712    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-var-lib-cni-bin\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.981114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980738    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9sbv\" (UniqueName: \"kubernetes.io/projected/434cf180-e16d-43be-9860-18996cf3c1b2-kube-api-access-m9sbv\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:42.981114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980766    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-sysconfig\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.981114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980799    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-sysctl-d\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.981114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980824    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-system-cni-dir\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.981114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980874    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/434cf180-e16d-43be-9860-18996cf3c1b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:42.981114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980911    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/434cf180-e16d-43be-9860-18996cf3c1b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:42.981114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.980941    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-var-lib-cni-multus\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.981114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981011    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-hostroot\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.981114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981047    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72686aad-8df7-4b7c-b650-e169bc50ac5a-iptables-alerter-script\") pod \"iptables-alerter-7jhdj\" (UID: \"72686aad-8df7-4b7c-b650-e169bc50ac5a\") " pod="openshift-network-operator/iptables-alerter-7jhdj"
Sep 16 21:22:42.981114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981083    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-kubernetes\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981119    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be456967-5d82-446a-be28-ded3400718b6-cni-binary-copy\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981145    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldjsf\" (UniqueName: \"kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf\") pod \"network-check-target-6wzrh\" (UID: \"40074060-f15e-450c-b11a-534e9ab5eb75\") " pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981186    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cba9022f-9608-47da-8766-4fd3f31909d4-agent-certs\") pod \"konnectivity-agent-khthp\" (UID: \"cba9022f-9608-47da-8766-4fd3f31909d4\") " pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981216    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72686aad-8df7-4b7c-b650-e169bc50ac5a-host-slash\") pod \"iptables-alerter-7jhdj\" (UID: \"72686aad-8df7-4b7c-b650-e169bc50ac5a\") " pod="openshift-network-operator/iptables-alerter-7jhdj"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981241    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/434cf180-e16d-43be-9860-18996cf3c1b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981291    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-modprobe-d\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981342    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-var-lib-kubelet\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981373    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16268f79-7553-4246-a98a-c086cc926e07-etc-tuned\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-multus-conf-dir\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981454    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cba9022f-9608-47da-8766-4fd3f31909d4-konnectivity-ca\") pod \"konnectivity-agent-khthp\" (UID: \"cba9022f-9608-47da-8766-4fd3f31909d4\") " pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981479    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/434cf180-e16d-43be-9860-18996cf3c1b2-cnibin\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:42.981694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.981526    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/434cf180-e16d-43be-9860-18996cf3c1b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:42.982352 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.982166    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 21:22:42.982352 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.982209    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 21:22:42.982352 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.982213    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 21:22:42.982911 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.982639    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 21:22:42.982911 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.982691    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9ftm7\""
Sep 16 21:22:42.982911 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.982751    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 21:22:42.982911 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.982895    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 21:22:42.983157 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.983053    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 21:22:42.983157 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.983064    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lhjks\""
Sep 16 21:22:42.983260 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.983164    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 21:22:42.983317 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:42.983288    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 21:22:43.011066 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.011041    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j49jg"
Sep 16 21:22:43.055677 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.055654    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:43.069873 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.069846    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 21:22:43.082563 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.082533    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72tfb\" (UniqueName: \"kubernetes.io/projected/72686aad-8df7-4b7c-b650-e169bc50ac5a-kube-api-access-72tfb\") pod \"iptables-alerter-7jhdj\" (UID: \"72686aad-8df7-4b7c-b650-e169bc50ac5a\") " pod="openshift-network-operator/iptables-alerter-7jhdj"
Sep 16 21:22:43.082858 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.082804    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-var-lib-kubelet\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.082954 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.082858    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-lib-modules\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.082954 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.082893    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16268f79-7553-4246-a98a-c086cc926e07-tmp\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.082954 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.082914    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-var-lib-kubelet\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.083116 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.082966    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spjfw\" (UniqueName: \"kubernetes.io/projected/16268f79-7553-4246-a98a-c086cc926e07-kube-api-access-spjfw\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.083116 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.082996    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9sbv\" (UniqueName: \"kubernetes.io/projected/434cf180-e16d-43be-9860-18996cf3c1b2-kube-api-access-m9sbv\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.083116 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083024    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7a5797dc-757d-4e24-8cd8-e37626fe3ff4-hosts-file\") pod \"node-resolver-j4dhm\" (UID: \"7a5797dc-757d-4e24-8cd8-e37626fe3ff4\") " pod="openshift-dns/node-resolver-j4dhm"
Sep 16 21:22:43.083116 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083049    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.083116 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083105    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-lib-modules\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.083405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083113    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-sysctl-d\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.083405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083162    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/434cf180-e16d-43be-9860-18996cf3c1b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.083405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/434cf180-e16d-43be-9860-18996cf3c1b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.083405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083223    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/150f4568-938b-47df-a346-09c3fd2e5dac-kubelet-config\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:43.083405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083242    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-sysctl-d\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.083405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083251    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:43.083405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083286    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-sys-fs\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.083405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083335    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-env-overrides\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.083405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083366    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-hostroot\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.083405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083389    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a5797dc-757d-4e24-8cd8-e37626fe3ff4-tmp-dir\") pod \"node-resolver-j4dhm\" (UID: \"7a5797dc-757d-4e24-8cd8-e37626fe3ff4\") " pod="openshift-dns/node-resolver-j4dhm"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083415    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083464    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083485    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-hostroot\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083524    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-var-lib-openvswitch\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083572    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-run-openvswitch\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083624    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-run-ovn\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083665    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be456967-5d82-446a-be28-ded3400718b6-cni-binary-copy\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083691    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjsf\" (UniqueName: \"kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf\") pod \"network-check-target-6wzrh\" (UID: \"40074060-f15e-450c-b11a-534e9ab5eb75\") " pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083795    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cba9022f-9608-47da-8766-4fd3f31909d4-agent-certs\") pod \"konnectivity-agent-khthp\" (UID: \"cba9022f-9608-47da-8766-4fd3f31909d4\") " pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083831    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72686aad-8df7-4b7c-b650-e169bc50ac5a-host-slash\") pod \"iptables-alerter-7jhdj\" (UID: \"72686aad-8df7-4b7c-b650-e169bc50ac5a\") " pod="openshift-network-operator/iptables-alerter-7jhdj"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083858    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/434cf180-e16d-43be-9860-18996cf3c1b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083885    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-run-systemd\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.083910    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16268f79-7553-4246-a98a-c086cc926e07-etc-tuned\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084057    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/434cf180-e16d-43be-9860-18996cf3c1b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084064    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/434cf180-e16d-43be-9860-18996cf3c1b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084081    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/434cf180-e16d-43be-9860-18996cf3c1b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.084959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084114    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72686aad-8df7-4b7c-b650-e169bc50ac5a-host-slash\") pod \"iptables-alerter-7jhdj\" (UID: \"72686aad-8df7-4b7c-b650-e169bc50ac5a\") " pod="openshift-network-operator/iptables-alerter-7jhdj"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-multus-conf-dir\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084146    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-multus-conf-dir\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084155    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084187    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9m4\" (UniqueName: \"kubernetes.io/projected/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-kube-api-access-4b9m4\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084214    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-run-netns\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.084230    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084242    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-ovn-node-metrics-cert\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.084318    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs podName:58b25e36-7281-4e13-9eb6-a0b3ff533a7e nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.584280655 +0000 UTC m=+2.969963721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs") pod "network-metrics-daemon-hn2j5" (UID: "58b25e36-7281-4e13-9eb6-a0b3ff533a7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084337    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-ovnkube-script-lib\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084365    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/434cf180-e16d-43be-9860-18996cf3c1b2-system-cni-dir\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084313    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be456967-5d82-446a-be28-ded3400718b6-cni-binary-copy\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084388    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/150f4568-938b-47df-a346-09c3fd2e5dac-dbus\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084431    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/434cf180-e16d-43be-9860-18996cf3c1b2-system-cni-dir\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084451    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-sys\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084476    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-etc-kubernetes\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084502    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-node-log\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.085738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-sys\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084529    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-multus-socket-dir-parent\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084556    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-run-k8s-cni-cncf-io\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084607    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-run-netns\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084615    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-run-k8s-cni-cncf-io\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084625    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-multus-socket-dir-parent\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-run-multus-certs\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084558    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-etc-kubernetes\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084719    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-kubelet\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084749    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-run-netns\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084782    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-run-multus-certs\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084812    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-run\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084842    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-host\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084848    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-run\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084881    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-var-lib-cni-bin\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084938    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-host\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlr9h\" (UniqueName: \"kubernetes.io/projected/7fa8e414-a609-4b5e-9343-1cda7ab8b199-kube-api-access-zlr9h\") pod \"node-ca-pgpsw\" (UID: \"7fa8e414-a609-4b5e-9343-1cda7ab8b199\") " pod="openshift-image-registry/node-ca-pgpsw"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084975    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-var-lib-cni-bin\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.086452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.084984    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-sysconfig\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085007    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-system-cni-dir\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085024    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-var-lib-cni-multus\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085044    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72686aad-8df7-4b7c-b650-e169bc50ac5a-iptables-alerter-script\") pod \"iptables-alerter-7jhdj\" (UID: \"72686aad-8df7-4b7c-b650-e169bc50ac5a\") " pod="openshift-network-operator/iptables-alerter-7jhdj"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085072    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fa8e414-a609-4b5e-9343-1cda7ab8b199-host\") pod \"node-ca-pgpsw\" (UID: \"7fa8e414-a609-4b5e-9343-1cda7ab8b199\") " pod="openshift-image-registry/node-ca-pgpsw"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085096    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-host-var-lib-cni-multus\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085095    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-socket-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085129    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-system-cni-dir\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085141    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-log-socket\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085141    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-sysconfig\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-kubernetes\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085221    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfr84\" (UniqueName: \"kubernetes.io/projected/7a5797dc-757d-4e24-8cd8-e37626fe3ff4-kube-api-access-mfr84\") pod \"node-resolver-j4dhm\" (UID: \"7a5797dc-757d-4e24-8cd8-e37626fe3ff4\") " pod="openshift-dns/node-resolver-j4dhm"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085249    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fa8e414-a609-4b5e-9343-1cda7ab8b199-serviceca\") pod \"node-ca-pgpsw\" (UID: \"7fa8e414-a609-4b5e-9343-1cda7ab8b199\") " pod="openshift-image-registry/node-ca-pgpsw"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085271    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-kubernetes\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085289    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-etc-selinux\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085315    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-cni-netd\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085340    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwrb\" (UniqueName: \"kubernetes.io/projected/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-kube-api-access-tfwrb\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.087118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-modprobe-d\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085395    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-var-lib-kubelet\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085421    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cba9022f-9608-47da-8766-4fd3f31909d4-konnectivity-ca\") pod \"konnectivity-agent-khthp\" (UID: \"cba9022f-9608-47da-8766-4fd3f31909d4\") " pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085465    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/434cf180-e16d-43be-9860-18996cf3c1b2-cnibin\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085484    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-modprobe-d\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085514    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/434cf180-e16d-43be-9860-18996cf3c1b2-cnibin\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085518    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-var-lib-kubelet\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085551    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/434cf180-e16d-43be-9860-18996cf3c1b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085580    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-device-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085607    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrq5d\" (UniqueName: \"kubernetes.io/projected/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-kube-api-access-qrq5d\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085612    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72686aad-8df7-4b7c-b650-e169bc50ac5a-iptables-alerter-script\") pod \"iptables-alerter-7jhdj\" (UID: \"72686aad-8df7-4b7c-b650-e169bc50ac5a\") " pod="openshift-network-operator/iptables-alerter-7jhdj"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085632    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-systemd-units\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-sysctl-conf\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085683    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-systemd\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085707    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-cnibin\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085733    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be456967-5d82-446a-be28-ded3400718b6-multus-daemon-config\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085762    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-registration-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.087905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085783    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-systemd\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085801    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-etc-openvswitch\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085865    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-cnibin\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085896    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16268f79-7553-4246-a98a-c086cc926e07-etc-sysctl-conf\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085900    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-cni-bin\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085943    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-ovnkube-config\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085972    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-multus-cni-dir\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.085999    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvxn9\" (UniqueName: \"kubernetes.io/projected/be456967-5d82-446a-be28-ded3400718b6-kube-api-access-pvxn9\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.086026    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-slash\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.086038    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cba9022f-9608-47da-8766-4fd3f31909d4-konnectivity-ca\") pod \"konnectivity-agent-khthp\" (UID: \"cba9022f-9608-47da-8766-4fd3f31909d4\") " pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.086052    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.086111    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/434cf180-e16d-43be-9860-18996cf3c1b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.086119    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be456967-5d82-446a-be28-ded3400718b6-multus-cni-dir\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.086206    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be456967-5d82-446a-be28-ded3400718b6-multus-daemon-config\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.087904    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16268f79-7553-4246-a98a-c086cc926e07-tmp\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.088687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.087922    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16268f79-7553-4246-a98a-c086cc926e07-etc-tuned\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.089325 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.089100    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cba9022f-9608-47da-8766-4fd3f31909d4-agent-certs\") pod \"konnectivity-agent-khthp\" (UID: \"cba9022f-9608-47da-8766-4fd3f31909d4\") " pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:22:43.097121 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.097084    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:43.097121 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.097110    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:43.097266 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.097132    2567 projected.go:196] Error preparing data for projected volume kube-api-access-ldjsf for pod openshift-network-diagnostics/network-check-target-6wzrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:43.097266 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.097231    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf podName:40074060-f15e-450c-b11a-534e9ab5eb75 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.597209071 +0000 UTC m=+2.982892141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ldjsf" (UniqueName: "kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf") pod "network-check-target-6wzrh" (UID: "40074060-f15e-450c-b11a-534e9ab5eb75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:43.099963 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.099904    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spjfw\" (UniqueName: \"kubernetes.io/projected/16268f79-7553-4246-a98a-c086cc926e07-kube-api-access-spjfw\") pod \"tuned-swwmb\" (UID: \"16268f79-7553-4246-a98a-c086cc926e07\") " pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.100209 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.100167    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9m4\" (UniqueName: \"kubernetes.io/projected/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-kube-api-access-4b9m4\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:43.100649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.100627    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvxn9\" (UniqueName: \"kubernetes.io/projected/be456967-5d82-446a-be28-ded3400718b6-kube-api-access-pvxn9\") pod \"multus-sdq4k\" (UID: \"be456967-5d82-446a-be28-ded3400718b6\") " pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.100739 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.100698    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9sbv\" (UniqueName: \"kubernetes.io/projected/434cf180-e16d-43be-9860-18996cf3c1b2-kube-api-access-m9sbv\") pod \"multus-additional-cni-plugins-s86bb\" (UID: \"434cf180-e16d-43be-9860-18996cf3c1b2\") " pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.101125 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.101088    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72tfb\" (UniqueName: \"kubernetes.io/projected/72686aad-8df7-4b7c-b650-e169bc50ac5a-kube-api-access-72tfb\") pod \"iptables-alerter-7jhdj\" (UID: \"72686aad-8df7-4b7c-b650-e169bc50ac5a\") " pod="openshift-network-operator/iptables-alerter-7jhdj"
Sep 16 21:22:43.188346 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188308    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-device-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.188547 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188353    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrq5d\" (UniqueName: \"kubernetes.io/projected/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-kube-api-access-qrq5d\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.188547 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188371    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-systemd-units\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188547 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188389    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-registration-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.188547 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188403    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-etc-openvswitch\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188547 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188440    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-cni-bin\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188547 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-systemd-units\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188600    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-registration-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188604    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-device-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188635    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-cni-bin\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-ovnkube-config\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188703    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-slash\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188713    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-etc-openvswitch\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188729    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188763    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7a5797dc-757d-4e24-8cd8-e37626fe3ff4-hosts-file\") pod \"node-resolver-j4dhm\" (UID: \"7a5797dc-757d-4e24-8cd8-e37626fe3ff4\") " pod="openshift-dns/node-resolver-j4dhm"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188768    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-slash\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188786    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188813    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/150f4568-938b-47df-a346-09c3fd2e5dac-kubelet-config\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188816    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188858    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7a5797dc-757d-4e24-8cd8-e37626fe3ff4-hosts-file\") pod \"node-resolver-j4dhm\" (UID: \"7a5797dc-757d-4e24-8cd8-e37626fe3ff4\") " pod="openshift-dns/node-resolver-j4dhm"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188868    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.188881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188863    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-sys-fs\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188913    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-env-overrides\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188918    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-sys-fs\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188928    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/150f4568-938b-47df-a346-09c3fd2e5dac-kubelet-config\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a5797dc-757d-4e24-8cd8-e37626fe3ff4-tmp-dir\") pod \"node-resolver-j4dhm\" (UID: \"7a5797dc-757d-4e24-8cd8-e37626fe3ff4\") " pod="openshift-dns/node-resolver-j4dhm"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188967    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.188995    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-var-lib-openvswitch\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189020    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-run-openvswitch\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.189025    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189064    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-var-lib-openvswitch\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.189104    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret podName:150f4568-938b-47df-a346-09c3fd2e5dac nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.689083319 +0000 UTC m=+3.074766398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret") pod "global-pull-secret-syncer-j92fr" (UID: "150f4568-938b-47df-a346-09c3fd2e5dac") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189124    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-run-openvswitch\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189125    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-run-ovn\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189148    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189160    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-run-ovn\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-run-systemd\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189206    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-run-systemd\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.189697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189228    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-run-netns\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189249    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a5797dc-757d-4e24-8cd8-e37626fe3ff4-tmp-dir\") pod \"node-resolver-j4dhm\" (UID: \"7a5797dc-757d-4e24-8cd8-e37626fe3ff4\") " pod="openshift-dns/node-resolver-j4dhm"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189257    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-ovn-node-metrics-cert\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-ovnkube-script-lib\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-run-netns\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189311    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/150f4568-938b-47df-a346-09c3fd2e5dac-dbus\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189332    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-env-overrides\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189341    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-node-log\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189384    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-node-log\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189386    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-kubelet\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlr9h\" (UniqueName: \"kubernetes.io/projected/7fa8e414-a609-4b5e-9343-1cda7ab8b199-kube-api-access-zlr9h\") pod \"node-ca-pgpsw\" (UID: \"7fa8e414-a609-4b5e-9343-1cda7ab8b199\") " pod="openshift-image-registry/node-ca-pgpsw"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189448    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fa8e414-a609-4b5e-9343-1cda7ab8b199-host\") pod \"node-ca-pgpsw\" (UID: \"7fa8e414-a609-4b5e-9343-1cda7ab8b199\") " pod="openshift-image-registry/node-ca-pgpsw"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189464    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/150f4568-938b-47df-a346-09c3fd2e5dac-dbus\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-socket-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189523    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-log-socket\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-kubelet\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189551    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfr84\" (UniqueName: \"kubernetes.io/projected/7a5797dc-757d-4e24-8cd8-e37626fe3ff4-kube-api-access-mfr84\") pod \"node-resolver-j4dhm\" (UID: \"7a5797dc-757d-4e24-8cd8-e37626fe3ff4\") " pod="openshift-dns/node-resolver-j4dhm"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189577    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-log-socket\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189585    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fa8e414-a609-4b5e-9343-1cda7ab8b199-serviceca\") pod \"node-ca-pgpsw\" (UID: \"7fa8e414-a609-4b5e-9343-1cda7ab8b199\") " pod="openshift-image-registry/node-ca-pgpsw"
Sep 16 21:22:43.190994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189524    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fa8e414-a609-4b5e-9343-1cda7ab8b199-host\") pod \"node-ca-pgpsw\" (UID: \"7fa8e414-a609-4b5e-9343-1cda7ab8b199\") " pod="openshift-image-registry/node-ca-pgpsw"
Sep 16 21:22:43.190994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-etc-selinux\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.190994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189638    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-socket-dir\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.190994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189640    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-cni-netd\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189714    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwrb\" (UniqueName: \"kubernetes.io/projected/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-kube-api-access-tfwrb\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189680    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-host-cni-netd\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189742    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-etc-selinux\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.190994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189824    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-ovnkube-script-lib\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189844    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-ovnkube-config\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.190994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.189988    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fa8e414-a609-4b5e-9343-1cda7ab8b199-serviceca\") pod \"node-ca-pgpsw\" (UID: \"7fa8e414-a609-4b5e-9343-1cda7ab8b199\") " pod="openshift-image-registry/node-ca-pgpsw"
Sep 16 21:22:43.191816 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.191789    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-ovn-node-metrics-cert\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.198212 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.198161    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrq5d\" (UniqueName: \"kubernetes.io/projected/34bd6f2e-48cf-4ccc-bd7b-52b683aec45d-kube-api-access-qrq5d\") pod \"aws-ebs-csi-driver-node-72ks7\" (UID: \"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.200513 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.200493    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlr9h\" (UniqueName: \"kubernetes.io/projected/7fa8e414-a609-4b5e-9343-1cda7ab8b199-kube-api-access-zlr9h\") pod \"node-ca-pgpsw\" (UID: \"7fa8e414-a609-4b5e-9343-1cda7ab8b199\") " pod="openshift-image-registry/node-ca-pgpsw"
Sep 16 21:22:43.204105 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.204085    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwrb\" (UniqueName: \"kubernetes.io/projected/f4887354-dd7a-4c42-8b29-50f06b2cb9b2-kube-api-access-tfwrb\") pod \"ovnkube-node-82sp7\" (UID: \"f4887354-dd7a-4c42-8b29-50f06b2cb9b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.204243 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.204220    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfr84\" (UniqueName: \"kubernetes.io/projected/7a5797dc-757d-4e24-8cd8-e37626fe3ff4-kube-api-access-mfr84\") pod \"node-resolver-j4dhm\" (UID: \"7a5797dc-757d-4e24-8cd8-e37626fe3ff4\") " pod="openshift-dns/node-resolver-j4dhm"
Sep 16 21:22:43.282953 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.282916    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7jhdj"
Sep 16 21:22:43.291093 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.291067    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-swwmb"
Sep 16 21:22:43.302044 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.302021    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s86bb"
Sep 16 21:22:43.310753 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.310725    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sdq4k"
Sep 16 21:22:43.318394 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.318373    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:22:43.327988 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.327968    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j4dhm"
Sep 16 21:22:43.335642 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.335624    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7"
Sep 16 21:22:43.345202 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.345185    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pgpsw"
Sep 16 21:22:43.351962 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.351946    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:22:43.592836 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.592751    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:43.593039 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.592953    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:43.593039 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.593033    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs podName:58b25e36-7281-4e13-9eb6-a0b3ff533a7e nodeName:}" failed. No retries permitted until 2026-09-16 21:22:44.593012237 +0000 UTC m=+3.978695324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs") pod "network-metrics-daemon-hn2j5" (UID: "58b25e36-7281-4e13-9eb6-a0b3ff533a7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:43.693903 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.693862    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:43.694069 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:43.693921    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjsf\" (UniqueName: \"kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf\") pod \"network-check-target-6wzrh\" (UID: \"40074060-f15e-450c-b11a-534e9ab5eb75\") " pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:43.694069 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.693991    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:43.694069 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.694045    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:43.694069 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.694063    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret podName:150f4568-938b-47df-a346-09c3fd2e5dac nodeName:}" failed. No retries permitted until 2026-09-16 21:22:44.694041515 +0000 UTC m=+4.079724591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret") pod "global-pull-secret-syncer-j92fr" (UID: "150f4568-938b-47df-a346-09c3fd2e5dac") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:43.694069 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.694064    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:43.694329 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.694081    2567 projected.go:196] Error preparing data for projected volume kube-api-access-ldjsf for pod openshift-network-diagnostics/network-check-target-6wzrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:43.694329 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:43.694117    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf podName:40074060-f15e-450c-b11a-534e9ab5eb75 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:44.694105965 +0000 UTC m=+4.079789044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldjsf" (UniqueName: "kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf") pod "network-check-target-6wzrh" (UID: "40074060-f15e-450c-b11a-534e9ab5eb75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:44.102272 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:44.102238    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:44.102770 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:44.102388    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:22:44.225563 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:44.225535    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe456967_5d82_446a_be28_ded3400718b6.slice/crio-ff6feace4c469667dd2d2aca073c8d05493d434eca097e8bdaa7240eab8c1ed3 WatchSource:0}: Error finding container ff6feace4c469667dd2d2aca073c8d05493d434eca097e8bdaa7240eab8c1ed3: Status 404 returned error can't find the container with id ff6feace4c469667dd2d2aca073c8d05493d434eca097e8bdaa7240eab8c1ed3
Sep 16 21:22:44.228108 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:44.228065    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72686aad_8df7_4b7c_b650_e169bc50ac5a.slice/crio-3dff2019f934e65639f7a5bc258e18e3e50ac46db3ec92ea052c7318bf64203d WatchSource:0}: Error finding container 3dff2019f934e65639f7a5bc258e18e3e50ac46db3ec92ea052c7318bf64203d: Status 404 returned error can't find the container with id 3dff2019f934e65639f7a5bc258e18e3e50ac46db3ec92ea052c7318bf64203d
Sep 16 21:22:44.231588 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:44.231560    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4887354_dd7a_4c42_8b29_50f06b2cb9b2.slice/crio-7b928e23e8cc8782420fd8d5dfd5927bca39827aa1ab130875235ad26864eb01 WatchSource:0}: Error finding container 7b928e23e8cc8782420fd8d5dfd5927bca39827aa1ab130875235ad26864eb01: Status 404 returned error can't find the container with id 7b928e23e8cc8782420fd8d5dfd5927bca39827aa1ab130875235ad26864eb01
Sep 16 21:22:44.232523 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:44.232499    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba9022f_9608_47da_8766_4fd3f31909d4.slice/crio-208f87d4b9cfc7777240e1a6c546cef25fb18fa92c9c989fb6219493dd16b699 WatchSource:0}: Error finding container 208f87d4b9cfc7777240e1a6c546cef25fb18fa92c9c989fb6219493dd16b699: Status 404 returned error can't find the container with id 208f87d4b9cfc7777240e1a6c546cef25fb18fa92c9c989fb6219493dd16b699
Sep 16 21:22:44.234549 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:44.233925    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa8e414_a609_4b5e_9343_1cda7ab8b199.slice/crio-845ed9f016f92fcd12753d638d3b5da46c67c8d3c80198f1b5cf37b279af6e09 WatchSource:0}: Error finding container 845ed9f016f92fcd12753d638d3b5da46c67c8d3c80198f1b5cf37b279af6e09: Status 404 returned error can't find the container with id 845ed9f016f92fcd12753d638d3b5da46c67c8d3c80198f1b5cf37b279af6e09
Sep 16 21:22:44.235122 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:44.235095    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bd6f2e_48cf_4ccc_bd7b_52b683aec45d.slice/crio-d32c3304eaa8f1ac031a6d870a6459b9cd769af7c69a979986637c9bc8f323a0 WatchSource:0}: Error finding container d32c3304eaa8f1ac031a6d870a6459b9cd769af7c69a979986637c9bc8f323a0: Status 404 returned error can't find the container with id d32c3304eaa8f1ac031a6d870a6459b9cd769af7c69a979986637c9bc8f323a0
Sep 16 21:22:44.236418 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:44.236362    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5797dc_757d_4e24_8cd8_e37626fe3ff4.slice/crio-850c463db125574e335634908733d66ddc5e1a309ad4f42c442c1c5d69432218 WatchSource:0}: Error finding container 850c463db125574e335634908733d66ddc5e1a309ad4f42c442c1c5d69432218: Status 404 returned error can't find the container with id 850c463db125574e335634908733d66ddc5e1a309ad4f42c442c1c5d69432218
Sep 16 21:22:44.237511 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:44.237367    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16268f79_7553_4246_a98a_c086cc926e07.slice/crio-5b0aaedde99a16531ceac63b7f2ce1fa15f94ddc4a84caaaf7f9c020f1d90001 WatchSource:0}: Error finding container 5b0aaedde99a16531ceac63b7f2ce1fa15f94ddc4a84caaaf7f9c020f1d90001: Status 404 returned error can't find the container with id 5b0aaedde99a16531ceac63b7f2ce1fa15f94ddc4a84caaaf7f9c020f1d90001
Sep 16 21:22:44.238587 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:22:44.238562    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434cf180_e16d_43be_9860_18996cf3c1b2.slice/crio-a977b766cb806dfce04b3360bf9c19685571d012547885c972059e383336e299 WatchSource:0}: Error finding container a977b766cb806dfce04b3360bf9c19685571d012547885c972059e383336e299: Status 404 returned error can't find the container with id a977b766cb806dfce04b3360bf9c19685571d012547885c972059e383336e299
Sep 16 21:22:44.602551 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:44.602525    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:44.602682 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:44.602630    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:44.602682 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:44.602679    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs podName:58b25e36-7281-4e13-9eb6-a0b3ff533a7e nodeName:}" failed. No retries permitted until 2026-09-16 21:22:46.602666096 +0000 UTC m=+5.988349160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs") pod "network-metrics-daemon-hn2j5" (UID: "58b25e36-7281-4e13-9eb6-a0b3ff533a7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:44.703032 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:44.703002    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:44.703184 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:44.703055    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjsf\" (UniqueName: \"kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf\") pod \"network-check-target-6wzrh\" (UID: \"40074060-f15e-450c-b11a-534e9ab5eb75\") " pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:44.703184 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:44.703146    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:44.703314 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:44.703202    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:44.703314 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:44.703219    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:44.703314 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:44.703229    2567 projected.go:196] Error preparing data for projected volume kube-api-access-ldjsf for pod openshift-network-diagnostics/network-check-target-6wzrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:44.703314 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:44.703233    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret podName:150f4568-938b-47df-a346-09c3fd2e5dac nodeName:}" failed. No retries permitted until 2026-09-16 21:22:46.703204528 +0000 UTC m=+6.088887623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret") pod "global-pull-secret-syncer-j92fr" (UID: "150f4568-938b-47df-a346-09c3fd2e5dac") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:44.703793 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:44.703774    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf podName:40074060-f15e-450c-b11a-534e9ab5eb75 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:46.703254371 +0000 UTC m=+6.088937434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldjsf" (UniqueName: "kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf") pod "network-check-target-6wzrh" (UID: "40074060-f15e-450c-b11a-534e9ab5eb75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:44.963788 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:44.963704    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:45.102768 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.102187    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:45.102768 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:45.102334    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:22:45.103931 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.103755    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:45.103931 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:45.103885    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:22:45.127940 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.127906    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal" event={"ID":"aef40cca25998ea7c58dc54dacd5bd30","Type":"ContainerStarted","Data":"dad8152c97583bc4728940bb05deb1c87c06d21fd519ac0e523ea32dff63ca28"}
Sep 16 21:22:45.131542 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.131499    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86bb" event={"ID":"434cf180-e16d-43be-9860-18996cf3c1b2","Type":"ContainerStarted","Data":"a977b766cb806dfce04b3360bf9c19685571d012547885c972059e383336e299"}
Sep 16 21:22:45.134313 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.134257    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-swwmb" event={"ID":"16268f79-7553-4246-a98a-c086cc926e07","Type":"ContainerStarted","Data":"5b0aaedde99a16531ceac63b7f2ce1fa15f94ddc4a84caaaf7f9c020f1d90001"}
Sep 16 21:22:45.141390 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.141359    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-khthp" event={"ID":"cba9022f-9608-47da-8766-4fd3f31909d4","Type":"ContainerStarted","Data":"208f87d4b9cfc7777240e1a6c546cef25fb18fa92c9c989fb6219493dd16b699"}
Sep 16 21:22:45.158390 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.157910    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-93.ec2.internal" event={"ID":"6bab6030418b48a95c5e7134d86b1cea","Type":"ContainerStarted","Data":"e0b8ef1b6633c43ec4f38939031c73f83249e93ca21bd274fbb568f243bcdd74"}
Sep 16 21:22:45.169006 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.168974    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j4dhm" event={"ID":"7a5797dc-757d-4e24-8cd8-e37626fe3ff4","Type":"ContainerStarted","Data":"850c463db125574e335634908733d66ddc5e1a309ad4f42c442c1c5d69432218"}
Sep 16 21:22:45.179243 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.179214    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7" event={"ID":"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d","Type":"ContainerStarted","Data":"d32c3304eaa8f1ac031a6d870a6459b9cd769af7c69a979986637c9bc8f323a0"}
Sep 16 21:22:45.187837 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.187809    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pgpsw" event={"ID":"7fa8e414-a609-4b5e-9343-1cda7ab8b199","Type":"ContainerStarted","Data":"845ed9f016f92fcd12753d638d3b5da46c67c8d3c80198f1b5cf37b279af6e09"}
Sep 16 21:22:45.193551 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.193524    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7" event={"ID":"f4887354-dd7a-4c42-8b29-50f06b2cb9b2","Type":"ContainerStarted","Data":"7b928e23e8cc8782420fd8d5dfd5927bca39827aa1ab130875235ad26864eb01"}
Sep 16 21:22:45.202128 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.202096    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7jhdj" event={"ID":"72686aad-8df7-4b7c-b650-e169bc50ac5a","Type":"ContainerStarted","Data":"3dff2019f934e65639f7a5bc258e18e3e50ac46db3ec92ea052c7318bf64203d"}
Sep 16 21:22:45.208611 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:45.208583    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sdq4k" event={"ID":"be456967-5d82-446a-be28-ded3400718b6","Type":"ContainerStarted","Data":"ff6feace4c469667dd2d2aca073c8d05493d434eca097e8bdaa7240eab8c1ed3"}
Sep 16 21:22:46.101841 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:46.101806    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:46.102014 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:46.101940    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:22:46.621899 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:46.621863    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:46.622387 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:46.622067    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:46.622387 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:46.622129    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs podName:58b25e36-7281-4e13-9eb6-a0b3ff533a7e nodeName:}" failed. No retries permitted until 2026-09-16 21:22:50.622111136 +0000 UTC m=+10.007794214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs") pod "network-metrics-daemon-hn2j5" (UID: "58b25e36-7281-4e13-9eb6-a0b3ff533a7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:46.723073 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:46.722231    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:46.723073 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:46.722289    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjsf\" (UniqueName: \"kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf\") pod \"network-check-target-6wzrh\" (UID: \"40074060-f15e-450c-b11a-534e9ab5eb75\") " pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:46.723073 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:46.722463    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:46.723073 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:46.722498    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:46.723073 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:46.722512    2567 projected.go:196] Error preparing data for projected volume kube-api-access-ldjsf for pod openshift-network-diagnostics/network-check-target-6wzrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:46.723073 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:46.722573    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf podName:40074060-f15e-450c-b11a-534e9ab5eb75 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:50.722552176 +0000 UTC m=+10.108235243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldjsf" (UniqueName: "kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf") pod "network-check-target-6wzrh" (UID: "40074060-f15e-450c-b11a-534e9ab5eb75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:46.723073 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:46.722967    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:46.723073 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:46.723020    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret podName:150f4568-938b-47df-a346-09c3fd2e5dac nodeName:}" failed. No retries permitted until 2026-09-16 21:22:50.723003116 +0000 UTC m=+10.108686196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret") pod "global-pull-secret-syncer-j92fr" (UID: "150f4568-938b-47df-a346-09c3fd2e5dac") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:47.107794 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:47.107760    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:47.107970 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:47.107901    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:22:47.108303 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:47.108259    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:47.108412 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:47.108357    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:22:48.102645 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:48.102613    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:48.103121 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:48.102739    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:22:49.102286 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:49.102054    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:49.102286 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:49.102166    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:49.102286 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:49.102175    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:22:49.102286 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:49.102241    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:22:50.102647 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:50.102075    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:50.102647 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:50.102214    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:22:50.658649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:50.658570    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:50.658840 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:50.658699    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:50.658840 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:50.658758    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs podName:58b25e36-7281-4e13-9eb6-a0b3ff533a7e nodeName:}" failed. No retries permitted until 2026-09-16 21:22:58.658744414 +0000 UTC m=+18.044427488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs") pod "network-metrics-daemon-hn2j5" (UID: "58b25e36-7281-4e13-9eb6-a0b3ff533a7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:50.760521 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:50.760478    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:50.760738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:50.760535    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjsf\" (UniqueName: \"kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf\") pod \"network-check-target-6wzrh\" (UID: \"40074060-f15e-450c-b11a-534e9ab5eb75\") " pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:50.760738 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:50.760721    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:50.760849 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:50.760742    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:50.760849 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:50.760754    2567 projected.go:196] Error preparing data for projected volume kube-api-access-ldjsf for pod openshift-network-diagnostics/network-check-target-6wzrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:50.760849 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:50.760818    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf podName:40074060-f15e-450c-b11a-534e9ab5eb75 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:58.760801871 +0000 UTC m=+18.146484949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldjsf" (UniqueName: "kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf") pod "network-check-target-6wzrh" (UID: "40074060-f15e-450c-b11a-534e9ab5eb75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:50.761255 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:50.761224    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:50.761362 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:50.761300    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret podName:150f4568-938b-47df-a346-09c3fd2e5dac nodeName:}" failed. No retries permitted until 2026-09-16 21:22:58.76127692 +0000 UTC m=+18.146959999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret") pod "global-pull-secret-syncer-j92fr" (UID: "150f4568-938b-47df-a346-09c3fd2e5dac") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:51.103643 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:51.103607    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:51.104198 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:51.103736    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:22:51.104256 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:51.104204    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:51.104358 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:51.104311    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:22:52.102075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:52.102042    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:52.102254 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:52.102170    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:22:53.102855 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:53.102822    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:53.103374 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:53.102977    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:22:53.103374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:53.103027    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:53.103374 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:53.103150    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:22:54.102611 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:54.102573    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:54.102783 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:54.102707    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:22:55.102468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:55.102439    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:55.102857 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:55.102564    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:22:55.102857 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:55.102620    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:55.102857 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:55.102742    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:22:56.101711 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:56.101679    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:56.101871 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:56.101801    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:22:57.102236 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:57.102200    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:57.102727 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:57.102209    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:57.102727 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:57.102344    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:22:57.102727 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:57.102425    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:22:58.102032 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:58.101995    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:58.102241 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:58.102154    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:22:58.722965 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:58.722934    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:58.723436 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:58.723095    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:58.723436 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:58.723170    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs podName:58b25e36-7281-4e13-9eb6-a0b3ff533a7e nodeName:}" failed. No retries permitted until 2026-09-16 21:23:14.723149241 +0000 UTC m=+34.108832325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs") pod "network-metrics-daemon-hn2j5" (UID: "58b25e36-7281-4e13-9eb6-a0b3ff533a7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:58.824208 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:58.824178    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:22:58.824208 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:58.824214    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjsf\" (UniqueName: \"kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf\") pod \"network-check-target-6wzrh\" (UID: \"40074060-f15e-450c-b11a-534e9ab5eb75\") " pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:58.824461 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:58.824345    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:58.824461 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:58.824408    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:58.824461 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:58.824420    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret podName:150f4568-938b-47df-a346-09c3fd2e5dac nodeName:}" failed. No retries permitted until 2026-09-16 21:23:14.824389378 +0000 UTC m=+34.210072464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret") pod "global-pull-secret-syncer-j92fr" (UID: "150f4568-938b-47df-a346-09c3fd2e5dac") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:58.824461 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:58.824426    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:58.824461 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:58.824440    2567 projected.go:196] Error preparing data for projected volume kube-api-access-ldjsf for pod openshift-network-diagnostics/network-check-target-6wzrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:58.824708 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:58.824491    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf podName:40074060-f15e-450c-b11a-534e9ab5eb75 nodeName:}" failed. No retries permitted until 2026-09-16 21:23:14.824475277 +0000 UTC m=+34.210158386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldjsf" (UniqueName: "kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf") pod "network-check-target-6wzrh" (UID: "40074060-f15e-450c-b11a-534e9ab5eb75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:59.101798 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:59.101715    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:22:59.101949 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:22:59.101718    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:22:59.101949 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:59.101855    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:22:59.101949 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:22:59.101931    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:23:00.102597 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:00.102564    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:00.103001 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:00.102676    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:23:01.103412 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:01.103367    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:01.103853 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:01.103488    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:23:01.103853 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:01.103549    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:01.103853 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:01.103665    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:23:02.102481 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:02.102453    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:02.102636 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:02.102570    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:23:03.103121 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.102694    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:03.103739 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.102709    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:03.103739 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:03.103246    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:23:03.103739 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:03.103290    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:23:03.251078 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.251047    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j4dhm" event={"ID":"7a5797dc-757d-4e24-8cd8-e37626fe3ff4","Type":"ContainerStarted","Data":"facc8fb4fad3cb579fc0030ecc7bc1dc7ca2b331d76a92a378c15e47beb95d91"}
Sep 16 21:23:03.252511 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.252486    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7" event={"ID":"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d","Type":"ContainerStarted","Data":"cfc47149f08a066821d8b41217611622e431cfca7fe5f50ab77a68f7881f3b03"}
Sep 16 21:23:03.253895 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.253874    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pgpsw" event={"ID":"7fa8e414-a609-4b5e-9343-1cda7ab8b199","Type":"ContainerStarted","Data":"b12257ba59a2097e32a79fe2de3ed07ebc96d5219d1bd5e6a4f174662206b4bd"}
Sep 16 21:23:03.256981 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.256961    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7" event={"ID":"f4887354-dd7a-4c42-8b29-50f06b2cb9b2","Type":"ContainerStarted","Data":"6d2fcd2c47f4f6222361f76493d48266416af691fc0be94feafbb894ef15bff0"}
Sep 16 21:23:03.257065 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.257025    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7" event={"ID":"f4887354-dd7a-4c42-8b29-50f06b2cb9b2","Type":"ContainerStarted","Data":"df4634fcde4e6ad047be8c27d73e5612abdb55c3d98bc57c202936f7590ba5c1"}
Sep 16 21:23:03.257105 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.257071    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7" event={"ID":"f4887354-dd7a-4c42-8b29-50f06b2cb9b2","Type":"ContainerStarted","Data":"76e70a11dca56be0e887d93d8f13e4455a34d812bc44e4110a30c697321881fe"}
Sep 16 21:23:03.257105 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.257102    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7" event={"ID":"f4887354-dd7a-4c42-8b29-50f06b2cb9b2","Type":"ContainerStarted","Data":"3f4fcc430ca0052e090cb8a0a9710bcd0a1b82c9d08f0ebdea75f662b8fa26ff"}
Sep 16 21:23:03.257167 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.257129    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7" event={"ID":"f4887354-dd7a-4c42-8b29-50f06b2cb9b2","Type":"ContainerStarted","Data":"6c015a66c1f97cc064185315110b37dbeefbcfaae99e1e19805d8b70667b513f"}
Sep 16 21:23:03.257167 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.257158    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7" event={"ID":"f4887354-dd7a-4c42-8b29-50f06b2cb9b2","Type":"ContainerStarted","Data":"1402174e376cc2d13cd0a4eb2efd5aba980eb0bffc8be5b0cdf0fec2e3865dec"}
Sep 16 21:23:03.258359 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.258338    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sdq4k" event={"ID":"be456967-5d82-446a-be28-ded3400718b6","Type":"ContainerStarted","Data":"86a5d10c3050d3d93f159444678a8d8ac9d555cb5bc3aee1568ca1818a40671b"}
Sep 16 21:23:03.259789 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.259773    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal_aef40cca25998ea7c58dc54dacd5bd30/setup/0.log"
Sep 16 21:23:03.259865 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.259805    2567 generic.go:356] "Generic (PLEG): container finished" podID="aef40cca25998ea7c58dc54dacd5bd30" containerID="dad8152c97583bc4728940bb05deb1c87c06d21fd519ac0e523ea32dff63ca28" exitCode=1
Sep 16 21:23:03.259914 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.259863    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal" event={"ID":"aef40cca25998ea7c58dc54dacd5bd30","Type":"ContainerDied","Data":"dad8152c97583bc4728940bb05deb1c87c06d21fd519ac0e523ea32dff63ca28"}
Sep 16 21:23:03.261336 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.261318    2567 generic.go:356] "Generic (PLEG): container finished" podID="434cf180-e16d-43be-9860-18996cf3c1b2" containerID="5216869503f295f421f7a63a58d8f94f05372e6189e0742bcfa778db44f193a9" exitCode=0
Sep 16 21:23:03.261387 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.261358    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86bb" event={"ID":"434cf180-e16d-43be-9860-18996cf3c1b2","Type":"ContainerDied","Data":"5216869503f295f421f7a63a58d8f94f05372e6189e0742bcfa778db44f193a9"}
Sep 16 21:23:03.262853 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.262764    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-swwmb" event={"ID":"16268f79-7553-4246-a98a-c086cc926e07","Type":"ContainerStarted","Data":"16f49ff5e7ac2eedea48a5d9c9ca7a30b89712d9211068d44212ed6ccb1f9e01"}
Sep 16 21:23:03.264045 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.264024    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-khthp" event={"ID":"cba9022f-9608-47da-8766-4fd3f31909d4","Type":"ContainerStarted","Data":"b29f822893681884195e052e7645ee1e224784addd50323bd6002601392dfbb4"}
Sep 16 21:23:03.265480 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.265439    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-93.ec2.internal" podStartSLOduration=21.265426943 podStartE2EDuration="21.265426943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:45.174034897 +0000 UTC m=+4.559718001" watchObservedRunningTime="2026-09-16 21:23:03.265426943 +0000 UTC m=+22.651110027"
Sep 16 21:23:03.278616 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.278576    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j4dhm" podStartSLOduration=4.140870312 podStartE2EDuration="22.27856464s" totalImagesPullingTime="18.137694328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:03.264953201 +0000 UTC m=+22.650636285" watchObservedRunningTime="2026-09-16 21:23:03.27856464 +0000 UTC m=+22.664247724"
Sep 16 21:23:03.278781 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.278761    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pgpsw" podStartSLOduration=4.166551344 podStartE2EDuration="22.278756935s" totalImagesPullingTime="18.112205591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:03.278306755 +0000 UTC m=+22.663989915" watchObservedRunningTime="2026-09-16 21:23:03.278756935 +0000 UTC m=+22.664440028"
Sep 16 21:23:03.309244 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.309199    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-swwmb" podStartSLOduration=4.171573508 podStartE2EDuration="22.309183124s" totalImagesPullingTime="18.137609616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:03.308270639 +0000 UTC m=+22.693953738" watchObservedRunningTime="2026-09-16 21:23:03.309183124 +0000 UTC m=+22.694866209"
Sep 16 21:23:03.326591 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.326551    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sdq4k" podStartSLOduration=3.973014817 podStartE2EDuration="22.32653857s" totalImagesPullingTime="18.353523753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:03.325900101 +0000 UTC m=+22.711583185" watchObservedRunningTime="2026-09-16 21:23:03.32653857 +0000 UTC m=+22.712221656"
Sep 16 21:23:03.361321 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.361275    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-khthp" podStartSLOduration=8.271315807 podStartE2EDuration="22.361260791s" totalImagesPullingTime="14.089944984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:03.361029613 +0000 UTC m=+22.746712696" watchObservedRunningTime="2026-09-16 21:23:03.361260791 +0000 UTC m=+22.746943875"
Sep 16 21:23:03.895725 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:03.895691    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 21:23:04.029225 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:04.029110    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T21:23:03.895710418Z","UUID":"95810f4e-78dd-49cc-9ccd-9b5b603e9c0a","Handler":null,"Name":"","Endpoint":""}
Sep 16 21:23:04.030492 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:04.030477    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 21:23:04.030587 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:04.030504    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 21:23:04.102160 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:04.102133    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:04.102292 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:04.102253    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:23:04.268268 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:04.268236    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7" event={"ID":"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d","Type":"ContainerStarted","Data":"943252c63f2e58d41a952d64423667cff7f0ca1246a9dffda6ef8afed672814c"}
Sep 16 21:23:04.269699 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:04.269674    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7jhdj" event={"ID":"72686aad-8df7-4b7c-b650-e169bc50ac5a","Type":"ContainerStarted","Data":"bc88383b1ced24ff8ce5ab9d6704f43fb693b5fd09ae66bbdaa95ff162670484"}
Sep 16 21:23:04.271465 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:04.271449    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal_aef40cca25998ea7c58dc54dacd5bd30/setup/0.log"
Sep 16 21:23:04.271567 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:04.271503    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal" event={"ID":"aef40cca25998ea7c58dc54dacd5bd30","Type":"ContainerStarted","Data":"1257965765b1ab861b5f39bafb429fae5ae6a45b62ef84f7c46f470ff7230ef9"}
Sep 16 21:23:04.283240 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:04.283173    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7jhdj" podStartSLOduration=5.136256902 podStartE2EDuration="23.283160512s" totalImagesPullingTime="18.14690361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:04.282550161 +0000 UTC m=+23.668233255" watchObservedRunningTime="2026-09-16 21:23:04.283160512 +0000 UTC m=+23.668843590"
Sep 16 21:23:04.627337 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:04.627055    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j49jg"
Sep 16 21:23:05.102004 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:05.101980    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:05.102111 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:05.101986    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:05.102170 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:05.102112    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:23:05.102170 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:05.102138    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:23:05.275889 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:05.275849    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7" event={"ID":"34bd6f2e-48cf-4ccc-bd7b-52b683aec45d","Type":"ContainerStarted","Data":"258aa77198acf44cfe3b6ea648651f6c8700311a523a60b8e06c9e649884e033"}
Sep 16 21:23:05.279879 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:05.279857    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7" event={"ID":"f4887354-dd7a-4c42-8b29-50f06b2cb9b2","Type":"ContainerStarted","Data":"3f7093bf1f06fdcce685425e7cbe233075be92b84df63b3eb5fd3b4f0758f161"}
Sep 16 21:23:05.294312 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:05.294268    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-72ks7" podStartSLOduration=4.00356834 podStartE2EDuration="24.294253873s" totalImagesPullingTime="20.290685533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:05.293571517 +0000 UTC m=+24.679254612" watchObservedRunningTime="2026-09-16 21:23:05.294253873 +0000 UTC m=+24.679936957"
Sep 16 21:23:05.640992 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:05.640962    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 21:18:04 +0000 UTC" deadline="2028-06-29 04:13:18.550494234 +0000 UTC"
Sep 16 21:23:05.640992 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:05.640992    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15630h50m12.909506705s"
Sep 16 21:23:05.813456 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:05.813429    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:23:06.101713 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:06.101680    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:06.101863 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:06.101782    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:23:06.283433 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:06.283390    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal_aef40cca25998ea7c58dc54dacd5bd30/setup/0.log"
Sep 16 21:23:06.283803 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:06.283442    2567 generic.go:356] "Generic (PLEG): container finished" podID="aef40cca25998ea7c58dc54dacd5bd30" containerID="1257965765b1ab861b5f39bafb429fae5ae6a45b62ef84f7c46f470ff7230ef9" exitCode=0
Sep 16 21:23:06.283803 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:06.283521    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal" event={"ID":"aef40cca25998ea7c58dc54dacd5bd30","Type":"ContainerDied","Data":"1257965765b1ab861b5f39bafb429fae5ae6a45b62ef84f7c46f470ff7230ef9"}
Sep 16 21:23:06.283803 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:06.283572    2567 scope.go:117] "RemoveContainer" containerID="dad8152c97583bc4728940bb05deb1c87c06d21fd519ac0e523ea32dff63ca28"
Sep 16 21:23:06.283954 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:06.283901    2567 scope.go:117] "RemoveContainer" containerID="dad8152c97583bc4728940bb05deb1c87c06d21fd519ac0e523ea32dff63ca28"
Sep 16 21:23:06.292564 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:06.292538    2567 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal_openshift-machine-config-operator_aef40cca25998ea7c58dc54dacd5bd30_0 in pod sandbox 5cf4684368fa08b1d749a17e3d92c8a4ef06ac5473e9da5abffe97b75a753976 from index: no such id: 'dad8152c97583bc4728940bb05deb1c87c06d21fd519ac0e523ea32dff63ca28'" containerID="dad8152c97583bc4728940bb05deb1c87c06d21fd519ac0e523ea32dff63ca28"
Sep 16 21:23:06.292632 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:06.292576    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dad8152c97583bc4728940bb05deb1c87c06d21fd519ac0e523ea32dff63ca28"} err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal_openshift-machine-config-operator_aef40cca25998ea7c58dc54dacd5bd30_0 in pod sandbox 5cf4684368fa08b1d749a17e3d92c8a4ef06ac5473e9da5abffe97b75a753976 from index: no such id: 'dad8152c97583bc4728940bb05deb1c87c06d21fd519ac0e523ea32dff63ca28'"
Sep 16 21:23:06.309086 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:06.309064    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:23:06.309579 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:06.309561    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:23:06.641801 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:06.641724    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 21:18:04 +0000 UTC" deadline="2028-04-28 13:23:47.156553082 +0000 UTC"
Sep 16 21:23:06.641801 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:06.641756    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14152h0m40.514800983s"
Sep 16 21:23:07.102047 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:07.102016    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:07.102229 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:07.102137    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:23:07.102463 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:07.102016    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:07.102463 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:07.102427    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:23:07.287532 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:07.287498    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal" event={"ID":"aef40cca25998ea7c58dc54dacd5bd30","Type":"ContainerStarted","Data":"7374dbf10535b745e78736fc6dc144f20877932d0d4d7735e28b0956c7c1e2de"}
Sep 16 21:23:07.288302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:07.288281    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-khthp"
Sep 16 21:23:07.308341 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:07.308293    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-93.ec2.internal" podStartSLOduration=25.3082761 podStartE2EDuration="25.3082761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:07.308263361 +0000 UTC m=+26.693946463" watchObservedRunningTime="2026-09-16 21:23:07.3082761 +0000 UTC m=+26.693959185"
Sep 16 21:23:08.101857 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:08.101824    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:08.102036 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:08.101920    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:23:08.293210 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:08.292951    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86bb" event={"ID":"434cf180-e16d-43be-9860-18996cf3c1b2","Type":"ContainerStarted","Data":"8c05c1c3237b1e0bc6a41d8e745dfa74eb6652b7db3af8b943921951a87abfbe"}
Sep 16 21:23:08.297817 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:08.297791    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7" event={"ID":"f4887354-dd7a-4c42-8b29-50f06b2cb9b2","Type":"ContainerStarted","Data":"fab68753a3754e9959ac0a0fb537b8468bdd4fb3cc503f6095c540b5e5e23ab7"}
Sep 16 21:23:08.342591 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:08.342531    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7" podStartSLOduration=8.195557502 podStartE2EDuration="26.342511437s" totalImagesPullingTime="18.146953935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:08.342233334 +0000 UTC m=+27.727916439" watchObservedRunningTime="2026-09-16 21:23:08.342511437 +0000 UTC m=+27.728194524"
Sep 16 21:23:09.102457 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:09.102428    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:09.102635 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:09.102543    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:23:09.102635 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:09.102599    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:09.102733 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:09.102704    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:23:09.301979 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:09.301947    2567 generic.go:356] "Generic (PLEG): container finished" podID="434cf180-e16d-43be-9860-18996cf3c1b2" containerID="8c05c1c3237b1e0bc6a41d8e745dfa74eb6652b7db3af8b943921951a87abfbe" exitCode=0
Sep 16 21:23:09.302432 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:09.302031    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86bb" event={"ID":"434cf180-e16d-43be-9860-18996cf3c1b2","Type":"ContainerDied","Data":"8c05c1c3237b1e0bc6a41d8e745dfa74eb6652b7db3af8b943921951a87abfbe"}
Sep 16 21:23:09.302505 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:09.302435    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:23:09.302505 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:09.302501    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:23:09.302610 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:09.302544    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:23:09.317281 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:09.317254    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:23:09.317679 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:09.317666    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:23:10.093590 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:10.093549    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6wzrh"]
Sep 16 21:23:10.096671 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:10.094222    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:10.096671 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:10.094464    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:23:10.098349 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:10.098319    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hn2j5"]
Sep 16 21:23:10.098489 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:10.098475    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:10.098609 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:10.098592    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:23:10.098802 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:10.098784    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j92fr"]
Sep 16 21:23:10.098872 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:10.098862    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:10.098956 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:10.098940    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:23:10.307079 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:10.306828    2567 generic.go:356] "Generic (PLEG): container finished" podID="434cf180-e16d-43be-9860-18996cf3c1b2" containerID="27d16a057b26449503877746436ea29fc520993ddcc5c3248fc3fac890dcc6d6" exitCode=0
Sep 16 21:23:10.307684 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:10.306904    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86bb" event={"ID":"434cf180-e16d-43be-9860-18996cf3c1b2","Type":"ContainerDied","Data":"27d16a057b26449503877746436ea29fc520993ddcc5c3248fc3fac890dcc6d6"}
Sep 16 21:23:11.311750 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:11.311721    2567 generic.go:356] "Generic (PLEG): container finished" podID="434cf180-e16d-43be-9860-18996cf3c1b2" containerID="b8dcd616d3cefb685f0a78e18611167a46b39b7fbeb0384464b7b13a00865ab5" exitCode=0
Sep 16 21:23:11.312075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:11.311803    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86bb" event={"ID":"434cf180-e16d-43be-9860-18996cf3c1b2","Type":"ContainerDied","Data":"b8dcd616d3cefb685f0a78e18611167a46b39b7fbeb0384464b7b13a00865ab5"}
Sep 16 21:23:12.102112 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:12.102035    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:12.102237 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:12.102154    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:12.102237 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:12.102164    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:23:12.102237 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:12.102179    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:12.102346 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:12.102241    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:23:12.102346 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:12.102296    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:23:14.102724 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:14.102696    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:14.102724 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:14.102720    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:14.103202 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:14.102720    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:14.103202 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:14.102806    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2j5" podUID="58b25e36-7281-4e13-9eb6-a0b3ff533a7e"
Sep 16 21:23:14.103202 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:14.102903    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j92fr" podUID="150f4568-938b-47df-a346-09c3fd2e5dac"
Sep 16 21:23:14.103202 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:14.102996    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wzrh" podUID="40074060-f15e-450c-b11a-534e9ab5eb75"
Sep 16 21:23:14.745237 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:14.745209    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:14.745520 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:14.745317    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:23:14.745520 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:14.745373    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs podName:58b25e36-7281-4e13-9eb6-a0b3ff533a7e nodeName:}" failed. No retries permitted until 2026-09-16 21:23:46.745354963 +0000 UTC m=+66.131038028 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs") pod "network-metrics-daemon-hn2j5" (UID: "58b25e36-7281-4e13-9eb6-a0b3ff533a7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:23:14.845599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:14.845570    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:14.845599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:14.845604    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjsf\" (UniqueName: \"kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf\") pod \"network-check-target-6wzrh\" (UID: \"40074060-f15e-450c-b11a-534e9ab5eb75\") " pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:14.845761 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:14.845707    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:23:14.845795 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:14.845771    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret podName:150f4568-938b-47df-a346-09c3fd2e5dac nodeName:}" failed. No retries permitted until 2026-09-16 21:23:46.845754488 +0000 UTC m=+66.231437553 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret") pod "global-pull-secret-syncer-j92fr" (UID: "150f4568-938b-47df-a346-09c3fd2e5dac") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:23:14.845795 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:14.845777    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:23:14.845795 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:14.845793    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:23:14.845896 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:14.845804    2567 projected.go:196] Error preparing data for projected volume kube-api-access-ldjsf for pod openshift-network-diagnostics/network-check-target-6wzrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:23:14.845896 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:14.845840    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf podName:40074060-f15e-450c-b11a-534e9ab5eb75 nodeName:}" failed. No retries permitted until 2026-09-16 21:23:46.845829761 +0000 UTC m=+66.231512838 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldjsf" (UniqueName: "kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf") pod "network-check-target-6wzrh" (UID: "40074060-f15e-450c-b11a-534e9ab5eb75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:23:15.395585 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.395555    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-93.ec2.internal" event="NodeReady"
Sep 16 21:23:15.396013 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.395716    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 21:23:15.446893 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.446862    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-98sxj"]
Sep 16 21:23:15.469524 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.469475    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6n7r5"]
Sep 16 21:23:15.469710 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.469608    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.472762 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.472689    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 21:23:15.475565 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.475020    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 21:23:15.475565 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.475136    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rcnph\""
Sep 16 21:23:15.486499 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.486480    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-98sxj"]
Sep 16 21:23:15.486610 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.486521    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6n7r5"]
Sep 16 21:23:15.486610 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.486544    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pft6k"]
Sep 16 21:23:15.486610 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.486551    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6n7r5"
Sep 16 21:23:15.489930 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.489907    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 21:23:15.490040 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.489948    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-77lfh\""
Sep 16 21:23:15.490040 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.489950    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 21:23:15.490040 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.489992    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 21:23:15.502738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.502716    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pft6k"]
Sep 16 21:23:15.502842 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.502830    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.504968 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.504950    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 21:23:15.505110 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.504997    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 21:23:15.505110 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.504997    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 21:23:15.505218 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.505134    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hr64t\""
Sep 16 21:23:15.505218 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.505137    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 21:23:15.551196 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.551170    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42fd9e51-d6c3-4173-a89a-32dc02b95364-metrics-tls\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.551314 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.551207    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fd9e51-d6c3-4173-a89a-32dc02b95364-tmp-dir\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.551314 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.551262    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85jr\" (UniqueName: \"kubernetes.io/projected/42fd9e51-d6c3-4173-a89a-32dc02b95364-kube-api-access-s85jr\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.551314 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.551299    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42fd9e51-d6c3-4173-a89a-32dc02b95364-config-volume\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.651963 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.651931    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61943deb-41b6-4b74-8eb3-e9812d5a0797-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.652108 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.651973    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61943deb-41b6-4b74-8eb3-e9812d5a0797-crio-socket\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.652108 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.652020    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61943deb-41b6-4b74-8eb3-e9812d5a0797-data-volume\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.652108 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.652048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42fd9e51-d6c3-4173-a89a-32dc02b95364-metrics-tls\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.652108 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.652074    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fd9e51-d6c3-4173-a89a-32dc02b95364-tmp-dir\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.652108 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.652094    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1843cef4-b35a-44e8-9106-c2e609602879-cert\") pod \"ingress-canary-6n7r5\" (UID: \"1843cef4-b35a-44e8-9106-c2e609602879\") " pod="openshift-ingress-canary/ingress-canary-6n7r5"
Sep 16 21:23:15.652359 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.652125    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61943deb-41b6-4b74-8eb3-e9812d5a0797-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.652359 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.652173    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85jr\" (UniqueName: \"kubernetes.io/projected/42fd9e51-d6c3-4173-a89a-32dc02b95364-kube-api-access-s85jr\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.652359 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.652212    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxh2\" (UniqueName: \"kubernetes.io/projected/1843cef4-b35a-44e8-9106-c2e609602879-kube-api-access-9pxh2\") pod \"ingress-canary-6n7r5\" (UID: \"1843cef4-b35a-44e8-9106-c2e609602879\") " pod="openshift-ingress-canary/ingress-canary-6n7r5"
Sep 16 21:23:15.652359 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.652255    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzd4c\" (UniqueName: \"kubernetes.io/projected/61943deb-41b6-4b74-8eb3-e9812d5a0797-kube-api-access-xzd4c\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.652359 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.652294    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42fd9e51-d6c3-4173-a89a-32dc02b95364-config-volume\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.652643 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.652483    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fd9e51-d6c3-4173-a89a-32dc02b95364-tmp-dir\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.652850 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.652826    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42fd9e51-d6c3-4173-a89a-32dc02b95364-config-volume\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.656875 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.656854    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42fd9e51-d6c3-4173-a89a-32dc02b95364-metrics-tls\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.661511 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.661487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85jr\" (UniqueName: \"kubernetes.io/projected/42fd9e51-d6c3-4173-a89a-32dc02b95364-kube-api-access-s85jr\") pod \"dns-default-98sxj\" (UID: \"42fd9e51-d6c3-4173-a89a-32dc02b95364\") " pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.753719 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.753686    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxh2\" (UniqueName: \"kubernetes.io/projected/1843cef4-b35a-44e8-9106-c2e609602879-kube-api-access-9pxh2\") pod \"ingress-canary-6n7r5\" (UID: \"1843cef4-b35a-44e8-9106-c2e609602879\") " pod="openshift-ingress-canary/ingress-canary-6n7r5"
Sep 16 21:23:15.753876 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.753739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzd4c\" (UniqueName: \"kubernetes.io/projected/61943deb-41b6-4b74-8eb3-e9812d5a0797-kube-api-access-xzd4c\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.753876 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.753769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61943deb-41b6-4b74-8eb3-e9812d5a0797-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.753876 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.753787    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61943deb-41b6-4b74-8eb3-e9812d5a0797-crio-socket\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.753876 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.753813    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61943deb-41b6-4b74-8eb3-e9812d5a0797-data-volume\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.753876 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.753837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1843cef4-b35a-44e8-9106-c2e609602879-cert\") pod \"ingress-canary-6n7r5\" (UID: \"1843cef4-b35a-44e8-9106-c2e609602879\") " pod="openshift-ingress-canary/ingress-canary-6n7r5"
Sep 16 21:23:15.753876 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.753862    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61943deb-41b6-4b74-8eb3-e9812d5a0797-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.754245 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.754199    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61943deb-41b6-4b74-8eb3-e9812d5a0797-crio-socket\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.754356 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.754317    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61943deb-41b6-4b74-8eb3-e9812d5a0797-data-volume\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.754466 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.754446    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61943deb-41b6-4b74-8eb3-e9812d5a0797-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.756487 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.756463    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61943deb-41b6-4b74-8eb3-e9812d5a0797-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.756680 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.756664    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1843cef4-b35a-44e8-9106-c2e609602879-cert\") pod \"ingress-canary-6n7r5\" (UID: \"1843cef4-b35a-44e8-9106-c2e609602879\") " pod="openshift-ingress-canary/ingress-canary-6n7r5"
Sep 16 21:23:15.760923 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.760869    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzd4c\" (UniqueName: \"kubernetes.io/projected/61943deb-41b6-4b74-8eb3-e9812d5a0797-kube-api-access-xzd4c\") pod \"insights-runtime-extractor-pft6k\" (UID: \"61943deb-41b6-4b74-8eb3-e9812d5a0797\") " pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:15.761172 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.761136    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxh2\" (UniqueName: \"kubernetes.io/projected/1843cef4-b35a-44e8-9106-c2e609602879-kube-api-access-9pxh2\") pod \"ingress-canary-6n7r5\" (UID: \"1843cef4-b35a-44e8-9106-c2e609602879\") " pod="openshift-ingress-canary/ingress-canary-6n7r5"
Sep 16 21:23:15.785976 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.785955    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:15.797064 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.797042    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6n7r5"
Sep 16 21:23:15.813277 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:15.813255    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pft6k"
Sep 16 21:23:16.102415 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:16.102328    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:16.102415 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:16.102359    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:16.102415 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:16.102328    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:16.104957 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:16.104939    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 21:23:16.105599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:16.105442    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hdj86\""
Sep 16 21:23:16.105599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:16.105463    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 21:23:16.105599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:16.105505    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 21:23:16.105816 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:16.105667    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 21:23:16.105816 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:16.105756    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8xpmt\""
Sep 16 21:23:17.170965 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.170748    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6n7r5"]
Sep 16 21:23:17.174647 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:17.174620    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1843cef4_b35a_44e8_9106_c2e609602879.slice/crio-73047f1924758b2f35ab13d08e64ddbdc9b72eea4a8e57ea4dc580121a6db9a3 WatchSource:0}: Error finding container 73047f1924758b2f35ab13d08e64ddbdc9b72eea4a8e57ea4dc580121a6db9a3: Status 404 returned error can't find the container with id 73047f1924758b2f35ab13d08e64ddbdc9b72eea4a8e57ea4dc580121a6db9a3
Sep 16 21:23:17.179139 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.179106    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-98sxj"]
Sep 16 21:23:17.183708 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:17.183668    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fd9e51_d6c3_4173_a89a_32dc02b95364.slice/crio-0d6f91fd017856cb00835dcf52db79a4475b3513a9d270957cb38fc5fe944a9d WatchSource:0}: Error finding container 0d6f91fd017856cb00835dcf52db79a4475b3513a9d270957cb38fc5fe944a9d: Status 404 returned error can't find the container with id 0d6f91fd017856cb00835dcf52db79a4475b3513a9d270957cb38fc5fe944a9d
Sep 16 21:23:17.183708 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.183680    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pft6k"]
Sep 16 21:23:17.187414 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:17.186762    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61943deb_41b6_4b74_8eb3_e9812d5a0797.slice/crio-6c9811d20e93eaaca909ebcc1237551c93a0e94f38b10658b443e55cd407ca20 WatchSource:0}: Error finding container 6c9811d20e93eaaca909ebcc1237551c93a0e94f38b10658b443e55cd407ca20: Status 404 returned error can't find the container with id 6c9811d20e93eaaca909ebcc1237551c93a0e94f38b10658b443e55cd407ca20
Sep 16 21:23:17.327315 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.327281    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6n7r5" event={"ID":"1843cef4-b35a-44e8-9106-c2e609602879","Type":"ContainerStarted","Data":"73047f1924758b2f35ab13d08e64ddbdc9b72eea4a8e57ea4dc580121a6db9a3"}
Sep 16 21:23:17.331211 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.331115    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86bb" event={"ID":"434cf180-e16d-43be-9860-18996cf3c1b2","Type":"ContainerStarted","Data":"b97207416bf06d4d1989003810b8470d742c9cef36e31d43af6f186da74cbbc4"}
Sep 16 21:23:17.332442 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.332411    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98sxj" event={"ID":"42fd9e51-d6c3-4173-a89a-32dc02b95364","Type":"ContainerStarted","Data":"0d6f91fd017856cb00835dcf52db79a4475b3513a9d270957cb38fc5fe944a9d"}
Sep 16 21:23:17.334314 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.334292    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"]
Sep 16 21:23:17.355026 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.354997    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pft6k" event={"ID":"61943deb-41b6-4b74-8eb3-e9812d5a0797","Type":"ContainerStarted","Data":"6c9811d20e93eaaca909ebcc1237551c93a0e94f38b10658b443e55cd407ca20"}
Sep 16 21:23:17.355291 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.355278    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"]
Sep 16 21:23:17.355431 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.355230    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.359302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.359089    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 21:23:17.359302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.359114    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 21:23:17.359302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.359162    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 21:23:17.359302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.359100    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 21:23:17.359302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.359217    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rd5jr\""
Sep 16 21:23:17.359302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.359097    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 21:23:17.465773 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.465738    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f62ceb82-fb55-410e-8c06-9798e256d099-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.465905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.465806    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f62ceb82-fb55-410e-8c06-9798e256d099-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.465966 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.465902    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f62ceb82-fb55-410e-8c06-9798e256d099-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.466017 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.465968    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2vcv\" (UniqueName: \"kubernetes.io/projected/f62ceb82-fb55-410e-8c06-9798e256d099-kube-api-access-j2vcv\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.567087 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.567052    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f62ceb82-fb55-410e-8c06-9798e256d099-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.567233 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.567100    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f62ceb82-fb55-410e-8c06-9798e256d099-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.567233 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.567151    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f62ceb82-fb55-410e-8c06-9798e256d099-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.567233 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.567201    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2vcv\" (UniqueName: \"kubernetes.io/projected/f62ceb82-fb55-410e-8c06-9798e256d099-kube-api-access-j2vcv\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.567918 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.567896    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f62ceb82-fb55-410e-8c06-9798e256d099-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.570197 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.570178    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f62ceb82-fb55-410e-8c06-9798e256d099-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.570294 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.570275    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f62ceb82-fb55-410e-8c06-9798e256d099-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.584466 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.584445    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2vcv\" (UniqueName: \"kubernetes.io/projected/f62ceb82-fb55-410e-8c06-9798e256d099-kube-api-access-j2vcv\") pod \"prometheus-operator-68bcd7f987-85f7w\" (UID: \"f62ceb82-fb55-410e-8c06-9798e256d099\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.665828 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.665802    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"
Sep 16 21:23:17.975302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:17.975191    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-85f7w"]
Sep 16 21:23:17.978889 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:17.978797    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62ceb82_fb55_410e_8c06_9798e256d099.slice/crio-55c3f7e5da866236161ad068cc2efd252e17032b0022b649284ac3a3bfc59253 WatchSource:0}: Error finding container 55c3f7e5da866236161ad068cc2efd252e17032b0022b649284ac3a3bfc59253: Status 404 returned error can't find the container with id 55c3f7e5da866236161ad068cc2efd252e17032b0022b649284ac3a3bfc59253
Sep 16 21:23:18.339392 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:18.339312    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w" event={"ID":"f62ceb82-fb55-410e-8c06-9798e256d099","Type":"ContainerStarted","Data":"55c3f7e5da866236161ad068cc2efd252e17032b0022b649284ac3a3bfc59253"}
Sep 16 21:23:18.341530 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:18.341318    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pft6k" event={"ID":"61943deb-41b6-4b74-8eb3-e9812d5a0797","Type":"ContainerStarted","Data":"b6e76b2058b59b7fe2b5b23aa72acd4598208d815d60cf8a812facf85282025a"}
Sep 16 21:23:18.344714 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:18.344687    2567 generic.go:356] "Generic (PLEG): container finished" podID="434cf180-e16d-43be-9860-18996cf3c1b2" containerID="b97207416bf06d4d1989003810b8470d742c9cef36e31d43af6f186da74cbbc4" exitCode=0
Sep 16 21:23:18.344825 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:18.344728    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86bb" event={"ID":"434cf180-e16d-43be-9860-18996cf3c1b2","Type":"ContainerDied","Data":"b97207416bf06d4d1989003810b8470d742c9cef36e31d43af6f186da74cbbc4"}
Sep 16 21:23:19.351471 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:19.351426    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pft6k" event={"ID":"61943deb-41b6-4b74-8eb3-e9812d5a0797","Type":"ContainerStarted","Data":"346975a5642148ed0a3d929c7d5db0b1ab147dad74879ef38f0e03f6efb19839"}
Sep 16 21:23:19.355557 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:19.355521    2567 generic.go:356] "Generic (PLEG): container finished" podID="434cf180-e16d-43be-9860-18996cf3c1b2" containerID="6261ebfc1d314a2d53f9f521d7e16d683d562efe9c6e3fd0a012dfca0581de32" exitCode=0
Sep 16 21:23:19.355678 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:19.355562    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86bb" event={"ID":"434cf180-e16d-43be-9860-18996cf3c1b2","Type":"ContainerDied","Data":"6261ebfc1d314a2d53f9f521d7e16d683d562efe9c6e3fd0a012dfca0581de32"}
Sep 16 21:23:22.365620 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.365576    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6n7r5" event={"ID":"1843cef4-b35a-44e8-9106-c2e609602879","Type":"ContainerStarted","Data":"5f90cd8121c12847291ea4811f837d5983e540bbcdaf6b3c32dc4265d81be44e"}
Sep 16 21:23:22.367371 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.367336    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w" event={"ID":"f62ceb82-fb55-410e-8c06-9798e256d099","Type":"ContainerStarted","Data":"8f908efc52354404d23abbf66ce258d26eca90cc56f1078e37f295e85c3396ea"}
Sep 16 21:23:22.367499 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.367406    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w" event={"ID":"f62ceb82-fb55-410e-8c06-9798e256d099","Type":"ContainerStarted","Data":"91cbc0c99943aafb7c4baf36c1af79ecdce6fca6a2a697d8c1a0f27061f56eb6"}
Sep 16 21:23:22.369375 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.369353    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pft6k" event={"ID":"61943deb-41b6-4b74-8eb3-e9812d5a0797","Type":"ContainerStarted","Data":"002b2f938bc97dfca2de06d22aa42262d47ca8a981f7872936f2036b017536cb"}
Sep 16 21:23:22.372416 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.372380    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86bb" event={"ID":"434cf180-e16d-43be-9860-18996cf3c1b2","Type":"ContainerStarted","Data":"86e67eacab1019fbe4fd32cfc3ce0fcab7efa6eb7299f0706dc832292169e2d7"}
Sep 16 21:23:22.373861 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.373842    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98sxj" event={"ID":"42fd9e51-d6c3-4173-a89a-32dc02b95364","Type":"ContainerStarted","Data":"a86684f7c8721acfc73bb581449e18cd176598fee0b4ab64a0493786b3b38af8"}
Sep 16 21:23:22.373951 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.373882    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98sxj" event={"ID":"42fd9e51-d6c3-4173-a89a-32dc02b95364","Type":"ContainerStarted","Data":"88c072f0986117a088fe138653ef259f76314834bcf27196ed5a7b1b95715cbe"}
Sep 16 21:23:22.374003 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.373975    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:22.381210 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.381177    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6n7r5" podStartSLOduration=3.114877717 podStartE2EDuration="7.381166979s" totalImagesPullingTime="4.266289262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:22.380732615 +0000 UTC m=+41.766415698" watchObservedRunningTime="2026-09-16 21:23:22.381166979 +0000 UTC m=+41.766850053"
Sep 16 21:23:22.398559 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.398525    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-85f7w" podStartSLOduration=1.8795521819999998 podStartE2EDuration="5.398512842s" totalImagesPullingTime="3.51896066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:22.397728498 +0000 UTC m=+41.783411584" watchObservedRunningTime="2026-09-16 21:23:22.398512842 +0000 UTC m=+41.784195923"
Sep 16 21:23:22.424299 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.424258    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s86bb" podStartSLOduration=11.102085823 podStartE2EDuration="41.424243549s" totalImagesPullingTime="30.322157726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:22.423208323 +0000 UTC m=+41.808891426" watchObservedRunningTime="2026-09-16 21:23:22.424243549 +0000 UTC m=+41.809926633"
Sep 16 21:23:22.442749 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.442712    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-98sxj" podStartSLOduration=3.155522378 podStartE2EDuration="7.44269787s" totalImagesPullingTime="4.287175492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:22.442491094 +0000 UTC m=+41.828174182" watchObservedRunningTime="2026-09-16 21:23:22.44269787 +0000 UTC m=+41.828380948"
Sep 16 21:23:22.458734 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:22.458690    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pft6k" podStartSLOduration=3.435980545 podStartE2EDuration="7.458674927s" totalImagesPullingTime="4.022694382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:23:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:22.457912892 +0000 UTC m=+41.843595977" watchObservedRunningTime="2026-09-16 21:23:22.458674927 +0000 UTC m=+41.844358010"
Sep 16 21:23:24.727278 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.727224    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"]
Sep 16 21:23:24.745937 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.745903    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qwrmj"]
Sep 16 21:23:24.746058 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.746029    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:24.748959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.748939    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 21:23:24.749625 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.749606    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 21:23:24.749723 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.749703    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 21:23:24.749783 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.749731    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f7zhx\""
Sep 16 21:23:24.767629 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.767610    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"]
Sep 16 21:23:24.767629 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.767624    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:24.769887 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.769869    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 21:23:24.769956 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.769910    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q9ngt\""
Sep 16 21:23:24.770174 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.770152    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 21:23:24.770256 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.770156    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 21:23:24.922574 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922545    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-accelerators-collector-config\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:24.922702 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922609    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:24.922702 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922682    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-wtmp\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:24.922802 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922710    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:24.922802 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922730    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/474831f6-ad89-4837-b240-e1db1f74c15e-root\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:24.922802 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922745    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/aa0bee59-8a71-49a2-bd47-837d1153c745-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:24.922802 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922762    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:24.922972 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/474831f6-ad89-4837-b240-e1db1f74c15e-sys\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:24.922972 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922855    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h9g9\" (UniqueName: \"kubernetes.io/projected/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-api-access-9h9g9\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:24.922972 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922885    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sszh5\" (UniqueName: \"kubernetes.io/projected/474831f6-ad89-4837-b240-e1db1f74c15e-kube-api-access-sszh5\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:24.922972 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922910    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa0bee59-8a71-49a2-bd47-837d1153c745-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:24.922972 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.922962    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:24.923117 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.923000    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-textfile\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:24.923117 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.923021    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-tls\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:24.923117 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:24.923042    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/474831f6-ad89-4837-b240-e1db1f74c15e-metrics-client-ca\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024029 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.023954    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sszh5\" (UniqueName: \"kubernetes.io/projected/474831f6-ad89-4837-b240-e1db1f74c15e-kube-api-access-sszh5\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024029 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.023988    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa0bee59-8a71-49a2-bd47-837d1153c745-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.024029 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.024294 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024039    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-textfile\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024294 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024056    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-tls\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024294 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024078    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/474831f6-ad89-4837-b240-e1db1f74c15e-metrics-client-ca\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024294 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024096    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-accelerators-collector-config\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024294 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024143    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.024543 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024428    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-wtmp\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024543 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024474    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024543 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024505    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/474831f6-ad89-4837-b240-e1db1f74c15e-root\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024543 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024531    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/aa0bee59-8a71-49a2-bd47-837d1153c745-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.024741 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024545    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-wtmp\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024741 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024557    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.024741 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024611    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/474831f6-ad89-4837-b240-e1db1f74c15e-sys\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024741 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024642    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h9g9\" (UniqueName: \"kubernetes.io/projected/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-api-access-9h9g9\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.024913 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024760    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/474831f6-ad89-4837-b240-e1db1f74c15e-metrics-client-ca\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024913 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024795    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa0bee59-8a71-49a2-bd47-837d1153c745-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.024913 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024433    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-textfile\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.024913 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024867    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/474831f6-ad89-4837-b240-e1db1f74c15e-sys\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.025065 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.024913    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/474831f6-ad89-4837-b240-e1db1f74c15e-root\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.025183 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.025160    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.025358 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.025184    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/aa0bee59-8a71-49a2-bd47-837d1153c745-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.025358 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.025228    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-accelerators-collector-config\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.027524 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.027499    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-tls\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.027524 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.027509    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/474831f6-ad89-4837-b240-e1db1f74c15e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.027649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.027620    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.031682 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.031662    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sszh5\" (UniqueName: \"kubernetes.io/projected/474831f6-ad89-4837-b240-e1db1f74c15e-kube-api-access-sszh5\") pod \"node-exporter-qwrmj\" (UID: \"474831f6-ad89-4837-b240-e1db1f74c15e\") " pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.031682 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.031679    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h9g9\" (UniqueName: \"kubernetes.io/projected/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-api-access-9h9g9\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.041320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.041294    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa0bee59-8a71-49a2-bd47-837d1153c745-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-kftsz\" (UID: \"aa0bee59-8a71-49a2-bd47-837d1153c745\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.054686 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.054670    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"
Sep 16 21:23:25.075515 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.075491    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qwrmj"
Sep 16 21:23:25.119165 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:25.119138    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474831f6_ad89_4837_b240_e1db1f74c15e.slice/crio-9202c03301ea06768fb5bbc5980f330bcdd38595a5b2355b2d0cd538071ce485 WatchSource:0}: Error finding container 9202c03301ea06768fb5bbc5980f330bcdd38595a5b2355b2d0cd538071ce485: Status 404 returned error can't find the container with id 9202c03301ea06768fb5bbc5980f330bcdd38595a5b2355b2d0cd538071ce485
Sep 16 21:23:25.385750 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.385670    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwrmj" event={"ID":"474831f6-ad89-4837-b240-e1db1f74c15e","Type":"ContainerStarted","Data":"9202c03301ea06768fb5bbc5980f330bcdd38595a5b2355b2d0cd538071ce485"}
Sep 16 21:23:25.424169 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.424140    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz"]
Sep 16 21:23:25.427558 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:25.427533    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0bee59_8a71_49a2_bd47_837d1153c745.slice/crio-4ba2f20e0f4321d0262a3e7e23214812ac8943f14735c3a501dcdba6f5511b15 WatchSource:0}: Error finding container 4ba2f20e0f4321d0262a3e7e23214812ac8943f14735c3a501dcdba6f5511b15: Status 404 returned error can't find the container with id 4ba2f20e0f4321d0262a3e7e23214812ac8943f14735c3a501dcdba6f5511b15
Sep 16 21:23:25.730457 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.729238    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 21:23:25.741688 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.741660    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.743989 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.743964    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 21:23:25.744133 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.744002    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 21:23:25.744133 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.743964    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 21:23:25.744296 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.744274    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 21:23:25.744458 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.744376    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nxgqm\""
Sep 16 21:23:25.744458 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.744415    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 21:23:25.744855 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.744518    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 21:23:25.744855 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.744706    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 21:23:25.745246 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.743964    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 21:23:25.746447 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.745878    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 21:23:25.747927 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.747856    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 21:23:25.831511 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831462    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.831511 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831511    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.831779 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831535    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x9pp\" (UniqueName: \"kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-kube-api-access-4x9pp\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.831779 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831592    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.831779 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831621    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.831779 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831646    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.831779 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831669    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-web-config\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.831779 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831685    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.831779 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831763    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-config-out\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.832104 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831807    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.832104 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831823    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.832104 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831855    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.832104 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.831884    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932141    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-config-out\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932315 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932315 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932315 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932223    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932315 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932243    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932315 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932258    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932315 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932282    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932642 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932464    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x9pp\" (UniqueName: \"kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-kube-api-access-4x9pp\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932642 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932511    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932642 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932558    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932642 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932587    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932851 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932629    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-web-config\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932851 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.932680    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.932953 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:25.932874    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-trusted-ca-bundle podName:f8535c85-0583-4eae-bb67-f700361da016 nodeName:}" failed. No retries permitted until 2026-09-16 21:23:26.432853174 +0000 UTC m=+45.818536242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f8535c85-0583-4eae-bb67-f700361da016") : configmap references non-existent config key: ca-bundle.crt
Sep 16 21:23:25.933512 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.933043    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.933512 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:25.933146    2567 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 21:23:25.933512 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:23:25.933199    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-main-tls podName:f8535c85-0583-4eae-bb67-f700361da016 nodeName:}" failed. No retries permitted until 2026-09-16 21:23:26.433181706 +0000 UTC m=+45.818864782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f8535c85-0583-4eae-bb67-f700361da016") : secret "alertmanager-main-tls" not found
Sep 16 21:23:25.933765 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.933703    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.935053 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.935009    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-config-out\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.935252 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.935210    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.935339 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.935315    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.935453 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.935416    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.935542 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.935524    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.935604 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.935588    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.935922 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.935903    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-web-config\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.936193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.936176    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:25.942733 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:25.942713    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x9pp\" (UniqueName: \"kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-kube-api-access-4x9pp\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:26.390010 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:26.389974    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz" event={"ID":"aa0bee59-8a71-49a2-bd47-837d1153c745","Type":"ContainerStarted","Data":"4ba2f20e0f4321d0262a3e7e23214812ac8943f14735c3a501dcdba6f5511b15"}
Sep 16 21:23:26.435636 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:26.435603    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:26.435636 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:26.435637    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:26.436350 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:26.436330    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:26.437953 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:26.437933    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:26.653135 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:26.653114    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:23:26.965682 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:26.965625    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 21:23:26.968603 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:26.968579    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8535c85_0583_4eae_bb67_f700361da016.slice/crio-e1b964ba4b5a8825fb9fcf7afe292df8602c540aa3e6ee67e814a26e7c6b2ff6 WatchSource:0}: Error finding container e1b964ba4b5a8825fb9fcf7afe292df8602c540aa3e6ee67e814a26e7c6b2ff6: Status 404 returned error can't find the container with id e1b964ba4b5a8825fb9fcf7afe292df8602c540aa3e6ee67e814a26e7c6b2ff6
Sep 16 21:23:27.395487 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.395434    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerStarted","Data":"e1b964ba4b5a8825fb9fcf7afe292df8602c540aa3e6ee67e814a26e7c6b2ff6"}
Sep 16 21:23:27.397319 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.397285    2567 generic.go:356] "Generic (PLEG): container finished" podID="474831f6-ad89-4837-b240-e1db1f74c15e" containerID="ddfd15838c693faac99e15af46bf7bae7dc6caf17ed51a1e4c7857d263f51890" exitCode=0
Sep 16 21:23:27.397463 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.397331    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwrmj" event={"ID":"474831f6-ad89-4837-b240-e1db1f74c15e","Type":"ContainerDied","Data":"ddfd15838c693faac99e15af46bf7bae7dc6caf17ed51a1e4c7857d263f51890"}
Sep 16 21:23:27.776196 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.776135    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-9bdff54c7-w748q"]
Sep 16 21:23:27.789966 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.789943    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.792078 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.792055    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 21:23:27.792192 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.792140    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m9kf5\""
Sep 16 21:23:27.792261 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.792225    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 21:23:27.792261 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.792239    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 21:23:27.792364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.792308    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 21:23:27.792364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.792335    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8c1ptaqqd176g\""
Sep 16 21:23:27.792539 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.792526    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 21:23:27.795275 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.795255    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9bdff54c7-w748q"]
Sep 16 21:23:27.844781 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.844749    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nw8w\" (UniqueName: \"kubernetes.io/projected/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-kube-api-access-2nw8w\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.844898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.844800    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.844898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.844818    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.844898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.844834    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-tls\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.844898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.844875    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-grpc-tls\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.845052 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.844924    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.845052 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.844960    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.845052 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.845004    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-metrics-client-ca\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.945515 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.945454    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nw8w\" (UniqueName: \"kubernetes.io/projected/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-kube-api-access-2nw8w\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.945632 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.945516    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.945632 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.945541    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.945632 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.945568    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-tls\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.945632 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.945590    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-grpc-tls\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.945850 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.945659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.945850 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.945719    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.945850 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.945763    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-metrics-client-ca\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.946638 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.946582    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-metrics-client-ca\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.948576 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.948540    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.948885 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.948842    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.949509 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.949241    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.949509 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.949425    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.950642 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.950620    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-thanos-querier-tls\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.950763 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.950701    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-secret-grpc-tls\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:27.954362 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:27.954330    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nw8w\" (UniqueName: \"kubernetes.io/projected/b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e-kube-api-access-2nw8w\") pod \"thanos-querier-9bdff54c7-w748q\" (UID: \"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e\") " pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:28.100571 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:28.100544    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:28.410327 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:28.410214    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwrmj" event={"ID":"474831f6-ad89-4837-b240-e1db1f74c15e","Type":"ContainerStarted","Data":"bde0e58e07d773650e4924d3c09aa513a07336ff10624335210ea5d30d9eb4cf"}
Sep 16 21:23:28.410327 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:28.410313    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwrmj" event={"ID":"474831f6-ad89-4837-b240-e1db1f74c15e","Type":"ContainerStarted","Data":"3ef5b6b881b0ae93a9b7df82755d6dee8cace74dac2287fb512822cfe8ec3e87"}
Sep 16 21:23:28.412764 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:28.412733    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz" event={"ID":"aa0bee59-8a71-49a2-bd47-837d1153c745","Type":"ContainerStarted","Data":"f9e482274a3cc7f0c87e3ef6874b0a42095cfd7ce6e9057304f677a66f3f6e22"}
Sep 16 21:23:28.412919 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:28.412779    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz" event={"ID":"aa0bee59-8a71-49a2-bd47-837d1153c745","Type":"ContainerStarted","Data":"63bd9773cf9e66460d54d40083818ab145d514511d40044e57ef81d122514e2d"}
Sep 16 21:23:28.412919 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:28.412800    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz" event={"ID":"aa0bee59-8a71-49a2-bd47-837d1153c745","Type":"ContainerStarted","Data":"28162c56514f0f9f2c6570f22c8d37bdc92e08d8b32456da1dceb28367fd8e63"}
Sep 16 21:23:28.438605 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:28.438577    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9bdff54c7-w748q"]
Sep 16 21:23:28.461233 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:28.461177    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qwrmj" podStartSLOduration=2.933843119 podStartE2EDuration="4.461163288s" totalImagesPullingTime="1.451900923s" totalInitContainerRuntime="75.419246ms" isStatefulPod=true podCreationTimestamp="2026-09-16 21:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:28.460196058 +0000 UTC m=+47.845879193" watchObservedRunningTime="2026-09-16 21:23:28.461163288 +0000 UTC m=+47.846846366"
Sep 16 21:23:28.620364 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:28.620334    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cc2ce0_5b8b_41a0_a670_a9a81d0f7f9e.slice/crio-1e72a38cb95058546d96603d4cfdfd59c4089b881b4b4b50ad4f7bd55f908388 WatchSource:0}: Error finding container 1e72a38cb95058546d96603d4cfdfd59c4089b881b4b4b50ad4f7bd55f908388: Status 404 returned error can't find the container with id 1e72a38cb95058546d96603d4cfdfd59c4089b881b4b4b50ad4f7bd55f908388
Sep 16 21:23:29.132773 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.132703    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-kftsz" podStartSLOduration=2.676560931 podStartE2EDuration="5.132683607s" totalImagesPullingTime="2.456122676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:28.506550137 +0000 UTC m=+47.892233249" watchObservedRunningTime="2026-09-16 21:23:29.132683607 +0000 UTC m=+48.518366691"
Sep 16 21:23:29.133426 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.133407    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6f848466bd-gqxfc"]
Sep 16 21:23:29.150560 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.150537    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.152874 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.152853    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f848466bd-gqxfc"]
Sep 16 21:23:29.152999 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.152984    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 21:23:29.153100 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.153079    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 21:23:29.153660 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.153624    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 21:23:29.153660 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.153638    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 21:23:29.153803 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.153660    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f1tapkblfobqs\""
Sep 16 21:23:29.153803 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.153723    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hnv88\""
Sep 16 21:23:29.154900 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.154879    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9179cf9f-ac35-4f83-94d1-2c170833c6cf-audit-log\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.154995 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.154907    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgb9b\" (UniqueName: \"kubernetes.io/projected/9179cf9f-ac35-4f83-94d1-2c170833c6cf-kube-api-access-bgb9b\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.154995 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.154949    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9179cf9f-ac35-4f83-94d1-2c170833c6cf-secret-metrics-server-tls\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.154995 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.154978    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9179cf9f-ac35-4f83-94d1-2c170833c6cf-secret-metrics-server-client-certs\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.155135 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.155004    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9179cf9f-ac35-4f83-94d1-2c170833c6cf-metrics-server-audit-profiles\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.155135 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.155022    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9179cf9f-ac35-4f83-94d1-2c170833c6cf-client-ca-bundle\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.155225 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.155133    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9179cf9f-ac35-4f83-94d1-2c170833c6cf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.256181 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.256115    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9179cf9f-ac35-4f83-94d1-2c170833c6cf-secret-metrics-server-tls\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.256181 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.256147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9179cf9f-ac35-4f83-94d1-2c170833c6cf-secret-metrics-server-client-certs\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.256181 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.256168    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9179cf9f-ac35-4f83-94d1-2c170833c6cf-metrics-server-audit-profiles\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.256434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.256184    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9179cf9f-ac35-4f83-94d1-2c170833c6cf-client-ca-bundle\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.256434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.256259    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9179cf9f-ac35-4f83-94d1-2c170833c6cf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.256434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.256314    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9179cf9f-ac35-4f83-94d1-2c170833c6cf-audit-log\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.256434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.256336    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgb9b\" (UniqueName: \"kubernetes.io/projected/9179cf9f-ac35-4f83-94d1-2c170833c6cf-kube-api-access-bgb9b\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.256702 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.256671    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9179cf9f-ac35-4f83-94d1-2c170833c6cf-audit-log\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.257014 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.256986    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9179cf9f-ac35-4f83-94d1-2c170833c6cf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.257126 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.257108    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9179cf9f-ac35-4f83-94d1-2c170833c6cf-metrics-server-audit-profiles\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.258670 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.258646    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9179cf9f-ac35-4f83-94d1-2c170833c6cf-secret-metrics-server-client-certs\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.259019 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.259002    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9179cf9f-ac35-4f83-94d1-2c170833c6cf-secret-metrics-server-tls\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.259079 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.259062    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9179cf9f-ac35-4f83-94d1-2c170833c6cf-client-ca-bundle\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.264318 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.264301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgb9b\" (UniqueName: \"kubernetes.io/projected/9179cf9f-ac35-4f83-94d1-2c170833c6cf-kube-api-access-bgb9b\") pod \"metrics-server-6f848466bd-gqxfc\" (UID: \"9179cf9f-ac35-4f83-94d1-2c170833c6cf\") " pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.417461 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.417427    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q" event={"ID":"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e","Type":"ContainerStarted","Data":"1e72a38cb95058546d96603d4cfdfd59c4089b881b4b4b50ad4f7bd55f908388"}
Sep 16 21:23:29.419528 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.419384    2567 generic.go:356] "Generic (PLEG): container finished" podID="f8535c85-0583-4eae-bb67-f700361da016" containerID="57f0f05f24e799ff95cdbc21ccf086c192fdce780e42fef5b3822c12faabc4f0" exitCode=0
Sep 16 21:23:29.419528 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.419435    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerDied","Data":"57f0f05f24e799ff95cdbc21ccf086c192fdce780e42fef5b3822c12faabc4f0"}
Sep 16 21:23:29.488508 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.488476    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:29.522508 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.522481    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"]
Sep 16 21:23:29.553018 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.552910    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"]
Sep 16 21:23:29.553018 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.553016    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"
Sep 16 21:23:29.560599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.558015    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 21:23:29.560599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.559908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzwf\" (UniqueName: \"kubernetes.io/projected/2612272b-ccf5-4e52-b710-afb85dff8c7a-kube-api-access-wqzwf\") pod \"monitoring-plugin-5d85778b9d-t5fcf\" (UID: \"2612272b-ccf5-4e52-b710-afb85dff8c7a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"
Sep 16 21:23:29.560599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.560201    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2612272b-ccf5-4e52-b710-afb85dff8c7a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-t5fcf\" (UID: \"2612272b-ccf5-4e52-b710-afb85dff8c7a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"
Sep 16 21:23:29.560599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.560242    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5vkrd\""
Sep 16 21:23:29.661104 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.661073    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzwf\" (UniqueName: \"kubernetes.io/projected/2612272b-ccf5-4e52-b710-afb85dff8c7a-kube-api-access-wqzwf\") pod \"monitoring-plugin-5d85778b9d-t5fcf\" (UID: \"2612272b-ccf5-4e52-b710-afb85dff8c7a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"
Sep 16 21:23:29.661272 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.661126    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2612272b-ccf5-4e52-b710-afb85dff8c7a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-t5fcf\" (UID: \"2612272b-ccf5-4e52-b710-afb85dff8c7a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"
Sep 16 21:23:29.663948 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.663918    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2612272b-ccf5-4e52-b710-afb85dff8c7a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-t5fcf\" (UID: \"2612272b-ccf5-4e52-b710-afb85dff8c7a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"
Sep 16 21:23:29.676282 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.676256    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzwf\" (UniqueName: \"kubernetes.io/projected/2612272b-ccf5-4e52-b710-afb85dff8c7a-kube-api-access-wqzwf\") pod \"monitoring-plugin-5d85778b9d-t5fcf\" (UID: \"2612272b-ccf5-4e52-b710-afb85dff8c7a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"
Sep 16 21:23:29.865266 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.865195    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f848466bd-gqxfc"]
Sep 16 21:23:29.868195 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:29.868160    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9179cf9f_ac35_4f83_94d1_2c170833c6cf.slice/crio-dc19f22bd6eadcf1f4907f83ff00d854ad0509cda1123876f2a833fd6f728741 WatchSource:0}: Error finding container dc19f22bd6eadcf1f4907f83ff00d854ad0509cda1123876f2a833fd6f728741: Status 404 returned error can't find the container with id dc19f22bd6eadcf1f4907f83ff00d854ad0509cda1123876f2a833fd6f728741
Sep 16 21:23:29.868325 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:29.868244    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"
Sep 16 21:23:30.175837 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:30.175762    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"]
Sep 16 21:23:30.179040 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:30.179014    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2612272b_ccf5_4e52_b710_afb85dff8c7a.slice/crio-a6cf2b44016a673e6165d586c817384e301d41b2211744413f479b94c6c4f71c WatchSource:0}: Error finding container a6cf2b44016a673e6165d586c817384e301d41b2211744413f479b94c6c4f71c: Status 404 returned error can't find the container with id a6cf2b44016a673e6165d586c817384e301d41b2211744413f479b94c6c4f71c
Sep 16 21:23:30.429826 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:30.429737    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf" event={"ID":"2612272b-ccf5-4e52-b710-afb85dff8c7a","Type":"ContainerStarted","Data":"a6cf2b44016a673e6165d586c817384e301d41b2211744413f479b94c6c4f71c"}
Sep 16 21:23:30.431186 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:30.431149    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc" event={"ID":"9179cf9f-ac35-4f83-94d1-2c170833c6cf","Type":"ContainerStarted","Data":"dc19f22bd6eadcf1f4907f83ff00d854ad0509cda1123876f2a833fd6f728741"}
Sep 16 21:23:32.379693 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:32.379654    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-98sxj"
Sep 16 21:23:32.441671 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:32.441631    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerStarted","Data":"43aa79e3fe568b2dba717d860ad1545fbfa8799b8bb3efa56af2b9a9c86e64f6"}
Sep 16 21:23:32.443819 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:32.443787    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q" event={"ID":"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e","Type":"ContainerStarted","Data":"dc9280a9bbdac8fc36922b16240f433ace6e7b4649341924ada883a5ab12fa27"}
Sep 16 21:23:32.443937 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:32.443847    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q" event={"ID":"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e","Type":"ContainerStarted","Data":"1be3f8a1e65b871d3a679459e1195a11566ccf4917f8863feaad0e5b6caa2447"}
Sep 16 21:23:33.450898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:33.450865    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q" event={"ID":"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e","Type":"ContainerStarted","Data":"3673198abe085b5d1b0b593d2e335efd7b38ac871f03918f757008a91d68a12c"}
Sep 16 21:23:33.454057 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:33.454029    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerStarted","Data":"e7efafa43252fa5e0df493ec39f1e5d7ff35d46f1ad2726e9b00155f0d27bc22"}
Sep 16 21:23:33.454150 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:33.454102    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerStarted","Data":"5c006d324c3f92ad3c5acd6d115bdf3200a3532b61605ad5c4840d0f6fc30813"}
Sep 16 21:23:33.454150 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:33.454144    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerStarted","Data":"51bfd8b0041640820cd93a6ffff3e4497a54961ea5c1107ab7f22fe6673c84d6"}
Sep 16 21:23:33.454234 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:33.454185    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerStarted","Data":"4132944ac85938718646abf056eb9baa0b9928b648fd7cf7e738b34df8db5b88"}
Sep 16 21:23:33.455528 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:33.455506    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf" event={"ID":"2612272b-ccf5-4e52-b710-afb85dff8c7a","Type":"ContainerStarted","Data":"be9b4d5792b33db4c1d2ba211cfb1aeb771058d9fb054d6b2994b97ce424400b"}
Sep 16 21:23:33.455873 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:33.455857    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"
Sep 16 21:23:33.457040 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:33.457019    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc" event={"ID":"9179cf9f-ac35-4f83-94d1-2c170833c6cf","Type":"ContainerStarted","Data":"326d7d23b52cc91eb28ca44215eb7415017547a56f236883dcc2e7aeb734a103"}
Sep 16 21:23:33.461042 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:33.461023    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf"
Sep 16 21:23:33.471707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:33.471666    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t5fcf" podStartSLOduration=1.555478218 podStartE2EDuration="4.471653512s" totalImagesPullingTime="2.916175294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:33.470648573 +0000 UTC m=+52.856331657" watchObservedRunningTime="2026-09-16 21:23:33.471653512 +0000 UTC m=+52.857336647"
Sep 16 21:23:33.509469 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:33.509431    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc" podStartSLOduration=1.28611058 podStartE2EDuration="4.509416083s" totalImagesPullingTime="3.223305503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:33.508311705 +0000 UTC m=+52.893994828" watchObservedRunningTime="2026-09-16 21:23:33.509416083 +0000 UTC m=+52.895099163"
Sep 16 21:23:34.464191 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:34.464157    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerStarted","Data":"309a948e31e9450643052d079f04168e6baeedd7cb7cc60776265c8d320f9b5b"}
Sep 16 21:23:34.467041 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:34.467019    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q" event={"ID":"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e","Type":"ContainerStarted","Data":"e03c06d1a77ef184068c4f49189b6b911038f296bf7dea1d519fad0136b11d85"}
Sep 16 21:23:34.467178 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:34.467083    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q" event={"ID":"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e","Type":"ContainerStarted","Data":"878cb5f036e8f1ef3b12c2a9b239307b73d9bb58541164d396f989159f7adef1"}
Sep 16 21:23:34.467178 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:34.467125    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q" event={"ID":"b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e","Type":"ContainerStarted","Data":"94f814447d29aa1bf55d97a255a722fbb295c976167aff179ea9f9aab624beef"}
Sep 16 21:23:34.507664 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:34.507608    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.394436067 podStartE2EDuration="9.507592461s" totalImagesPullingTime="5.09649013s" totalInitContainerRuntime="16.666264ms" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:34.505722143 +0000 UTC m=+53.891405253" watchObservedRunningTime="2026-09-16 21:23:34.507592461 +0000 UTC m=+53.893275545"
Sep 16 21:23:34.531981 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:34.531924    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q" podStartSLOduration=3.499380339 podStartE2EDuration="7.531901875s" totalImagesPullingTime="4.032521536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:34.528697266 +0000 UTC m=+53.914380368" watchObservedRunningTime="2026-09-16 21:23:34.531901875 +0000 UTC m=+53.917584960"
Sep 16 21:23:35.471035 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:35.471000    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:36.480340 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:36.480309    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-9bdff54c7-w748q"
Sep 16 21:23:41.322824 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:41.322796    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82sp7"
Sep 16 21:23:46.803472 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:46.803439    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:46.805793 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:46.805770    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 21:23:46.816693 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:46.816669    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b25e36-7281-4e13-9eb6-a0b3ff533a7e-metrics-certs\") pod \"network-metrics-daemon-hn2j5\" (UID: \"58b25e36-7281-4e13-9eb6-a0b3ff533a7e\") " pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:46.904468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:46.904443    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:46.904573 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:46.904476    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjsf\" (UniqueName: \"kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf\") pod \"network-check-target-6wzrh\" (UID: \"40074060-f15e-450c-b11a-534e9ab5eb75\") " pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:46.906781 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:46.906763    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 21:23:46.906847 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:46.906763    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 21:23:46.916477 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:46.916459    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 21:23:46.916825 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:46.916802    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/150f4568-938b-47df-a346-09c3fd2e5dac-original-pull-secret\") pod \"global-pull-secret-syncer-j92fr\" (UID: \"150f4568-938b-47df-a346-09c3fd2e5dac\") " pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:46.927070 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:46.927049    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldjsf\" (UniqueName: \"kubernetes.io/projected/40074060-f15e-450c-b11a-534e9ab5eb75-kube-api-access-ldjsf\") pod \"network-check-target-6wzrh\" (UID: \"40074060-f15e-450c-b11a-534e9ab5eb75\") " pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:47.014323 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:47.014303    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j92fr"
Sep 16 21:23:47.022496 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:47.022480    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8xpmt\""
Sep 16 21:23:47.027132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:47.027111    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hdj86\""
Sep 16 21:23:47.030722 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:47.030709    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2j5"
Sep 16 21:23:47.035888 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:47.035878    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:47.337526 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:47.337453    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j92fr"]
Sep 16 21:23:47.341120 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:47.341096    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150f4568_938b_47df_a346_09c3fd2e5dac.slice/crio-ded44a98c969055e538710f28bc575162e584a2a9d9635d39d148eb1d95ba2bd WatchSource:0}: Error finding container ded44a98c969055e538710f28bc575162e584a2a9d9635d39d148eb1d95ba2bd: Status 404 returned error can't find the container with id ded44a98c969055e538710f28bc575162e584a2a9d9635d39d148eb1d95ba2bd
Sep 16 21:23:47.357324 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:47.357302    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hn2j5"]
Sep 16 21:23:47.360740 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:47.360718    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b25e36_7281_4e13_9eb6_a0b3ff533a7e.slice/crio-e922551a3e977965992461241a5ac8038c530fad57e341e8d51cfe93b673c66f WatchSource:0}: Error finding container e922551a3e977965992461241a5ac8038c530fad57e341e8d51cfe93b673c66f: Status 404 returned error can't find the container with id e922551a3e977965992461241a5ac8038c530fad57e341e8d51cfe93b673c66f
Sep 16 21:23:47.387629 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:47.387609    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6wzrh"]
Sep 16 21:23:47.390058 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:23:47.390039    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40074060_f15e_450c_b11a_534e9ab5eb75.slice/crio-f6488c514a74d5ccee8b4c888285c94627a7284614e8773992c7a5edbe41ac48 WatchSource:0}: Error finding container f6488c514a74d5ccee8b4c888285c94627a7284614e8773992c7a5edbe41ac48: Status 404 returned error can't find the container with id f6488c514a74d5ccee8b4c888285c94627a7284614e8773992c7a5edbe41ac48
Sep 16 21:23:47.520614 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:47.520585    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j92fr" event={"ID":"150f4568-938b-47df-a346-09c3fd2e5dac","Type":"ContainerStarted","Data":"ded44a98c969055e538710f28bc575162e584a2a9d9635d39d148eb1d95ba2bd"}
Sep 16 21:23:47.521727 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:47.521703    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6wzrh" event={"ID":"40074060-f15e-450c-b11a-534e9ab5eb75","Type":"ContainerStarted","Data":"f6488c514a74d5ccee8b4c888285c94627a7284614e8773992c7a5edbe41ac48"}
Sep 16 21:23:47.522850 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:47.522831    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn2j5" event={"ID":"58b25e36-7281-4e13-9eb6-a0b3ff533a7e","Type":"ContainerStarted","Data":"e922551a3e977965992461241a5ac8038c530fad57e341e8d51cfe93b673c66f"}
Sep 16 21:23:48.532799 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:48.532688    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn2j5" event={"ID":"58b25e36-7281-4e13-9eb6-a0b3ff533a7e","Type":"ContainerStarted","Data":"56c30d16bc8724435c3930154c19329bb136bb05a91ff9b539c251fc19bb348e"}
Sep 16 21:23:49.489096 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:49.489048    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:49.489282 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:49.489125    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:23:49.540102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:49.540044    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn2j5" event={"ID":"58b25e36-7281-4e13-9eb6-a0b3ff533a7e","Type":"ContainerStarted","Data":"ca90701c0625cf5c5ee4090d89233d3479470a9e8b357e3c16a369b0bb198670"}
Sep 16 21:23:49.558643 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:49.558574    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hn2j5" podStartSLOduration=67.510911018 podStartE2EDuration="1m8.558554539s" totalImagesPullingTime="1.047643521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:49.557289216 +0000 UTC m=+68.942972319" watchObservedRunningTime="2026-09-16 21:23:49.558554539 +0000 UTC m=+68.944237625"
Sep 16 21:23:52.557643 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:52.557609    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j92fr" event={"ID":"150f4568-938b-47df-a346-09c3fd2e5dac","Type":"ContainerStarted","Data":"f0ddf431f1949080fce6a73775874fb7568373d12d02bf771805dfddad74a72a"}
Sep 16 21:23:52.563228 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:52.563194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6wzrh" event={"ID":"40074060-f15e-450c-b11a-534e9ab5eb75","Type":"ContainerStarted","Data":"1a20946131bcbf30a95097ffafd8bea88a2d2b0d189465047d192200edd665e6"}
Sep 16 21:23:52.563343 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:52.563277    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:23:52.573840 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:52.573798    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j92fr" podStartSLOduration=67.042420219 podStartE2EDuration="1m11.573672392s" totalImagesPullingTime="4.531252173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:52.57365929 +0000 UTC m=+71.959342374" watchObservedRunningTime="2026-09-16 21:23:52.573672392 +0000 UTC m=+71.959355470"
Sep 16 21:23:52.586737 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:23:52.586696    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6wzrh" podStartSLOduration=67.107672666 podStartE2EDuration="1m11.586682435s" totalImagesPullingTime="4.479009769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:52.585789947 +0000 UTC m=+71.971473044" watchObservedRunningTime="2026-09-16 21:23:52.586682435 +0000 UTC m=+71.972365518"
Sep 16 21:24:09.494383 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:09.494273    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:24:09.498006 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:09.497982    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6f848466bd-gqxfc"
Sep 16 21:24:23.569754 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:23.569722    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6wzrh"
Sep 16 21:24:45.073894 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.073854    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 21:24:45.074471 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.074414    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="alertmanager" containerID="cri-o://43aa79e3fe568b2dba717d860ad1545fbfa8799b8bb3efa56af2b9a9c86e64f6" gracePeriod=120
Sep 16 21:24:45.074804 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.074752    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy" containerID="cri-o://5c006d324c3f92ad3c5acd6d115bdf3200a3532b61605ad5c4840d0f6fc30813" gracePeriod=120
Sep 16 21:24:45.074922 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.074895    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy-web" containerID="cri-o://51bfd8b0041640820cd93a6ffff3e4497a54961ea5c1107ab7f22fe6673c84d6" gracePeriod=120
Sep 16 21:24:45.074999 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.074933    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="prom-label-proxy" containerID="cri-o://309a948e31e9450643052d079f04168e6baeedd7cb7cc60776265c8d320f9b5b" gracePeriod=120
Sep 16 21:24:45.075058 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.075041    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="config-reloader" containerID="cri-o://4132944ac85938718646abf056eb9baa0b9928b648fd7cf7e738b34df8db5b88" gracePeriod=120
Sep 16 21:24:45.075145 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.075021    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy-metric" containerID="cri-o://e7efafa43252fa5e0df493ec39f1e5d7ff35d46f1ad2726e9b00155f0d27bc22" gracePeriod=120
Sep 16 21:24:45.767736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.767702    2567 generic.go:356] "Generic (PLEG): container finished" podID="f8535c85-0583-4eae-bb67-f700361da016" containerID="309a948e31e9450643052d079f04168e6baeedd7cb7cc60776265c8d320f9b5b" exitCode=0
Sep 16 21:24:45.767736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.767727    2567 generic.go:356] "Generic (PLEG): container finished" podID="f8535c85-0583-4eae-bb67-f700361da016" containerID="5c006d324c3f92ad3c5acd6d115bdf3200a3532b61605ad5c4840d0f6fc30813" exitCode=0
Sep 16 21:24:45.767736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.767735    2567 generic.go:356] "Generic (PLEG): container finished" podID="f8535c85-0583-4eae-bb67-f700361da016" containerID="4132944ac85938718646abf056eb9baa0b9928b648fd7cf7e738b34df8db5b88" exitCode=0
Sep 16 21:24:45.767736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.767741    2567 generic.go:356] "Generic (PLEG): container finished" podID="f8535c85-0583-4eae-bb67-f700361da016" containerID="43aa79e3fe568b2dba717d860ad1545fbfa8799b8bb3efa56af2b9a9c86e64f6" exitCode=0
Sep 16 21:24:45.767999 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.767772    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerDied","Data":"309a948e31e9450643052d079f04168e6baeedd7cb7cc60776265c8d320f9b5b"}
Sep 16 21:24:45.767999 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.767836    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerDied","Data":"5c006d324c3f92ad3c5acd6d115bdf3200a3532b61605ad5c4840d0f6fc30813"}
Sep 16 21:24:45.767999 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.767862    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerDied","Data":"4132944ac85938718646abf056eb9baa0b9928b648fd7cf7e738b34df8db5b88"}
Sep 16 21:24:45.767999 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:45.767887    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerDied","Data":"43aa79e3fe568b2dba717d860ad1545fbfa8799b8bb3efa56af2b9a9c86e64f6"}
Sep 16 21:24:46.775258 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.775229    2567 generic.go:356] "Generic (PLEG): container finished" podID="f8535c85-0583-4eae-bb67-f700361da016" containerID="e7efafa43252fa5e0df493ec39f1e5d7ff35d46f1ad2726e9b00155f0d27bc22" exitCode=0
Sep 16 21:24:46.775258 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.775254    2567 generic.go:356] "Generic (PLEG): container finished" podID="f8535c85-0583-4eae-bb67-f700361da016" containerID="51bfd8b0041640820cd93a6ffff3e4497a54961ea5c1107ab7f22fe6673c84d6" exitCode=0
Sep 16 21:24:46.775650 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.775297    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerDied","Data":"e7efafa43252fa5e0df493ec39f1e5d7ff35d46f1ad2726e9b00155f0d27bc22"}
Sep 16 21:24:46.775650 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.775375    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerDied","Data":"51bfd8b0041640820cd93a6ffff3e4497a54961ea5c1107ab7f22fe6673c84d6"}
Sep 16 21:24:46.812936 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.812915    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:46.838619 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.838594    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-web-config\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.838753 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.838627    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-main-db\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.838753 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.838654    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-web\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.838753 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.838671    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-main-tls\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.838753 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.838692    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x9pp\" (UniqueName: \"kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-kube-api-access-4x9pp\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.838753 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.838713    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-config-out\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.839002 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.838832    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-trusted-ca-bundle\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.839002 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.838903    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-config-volume\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.839002 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.838931    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-tls-assets\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.839002 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.838985    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.839448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.839013    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-cluster-tls-config\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.839448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.839054    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-metrics-client-ca\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.839448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.839354    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:24:46.839448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.839373    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f8535c85-0583-4eae-bb67-f700361da016\" (UID: \"f8535c85-0583-4eae-bb67-f700361da016\") "
Sep 16 21:24:46.839678 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.839659    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:46.839737 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.839685    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-main-db\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.840804 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.840757    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:46.842994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.842963    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:46.843347 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.843306    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:24:46.843903 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.843772    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-kube-api-access-4x9pp" (OuterVolumeSpecName: "kube-api-access-4x9pp") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "kube-api-access-4x9pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:24:46.843903 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.843844    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-config-volume" (OuterVolumeSpecName: "config-volume") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:46.844079 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.844021    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:46.844240 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.844217    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-config-out" (OuterVolumeSpecName: "config-out") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:24:46.844634 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.844608    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:46.845375 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.845350    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:46.850664 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.850635    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:46.856178 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.856158    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-web-config" (OuterVolumeSpecName: "web-config") pod "f8535c85-0583-4eae-bb67-f700361da016" (UID: "f8535c85-0583-4eae-bb67-f700361da016"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:46.939982 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.939953    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.939982 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.939979    2567 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-cluster-tls-config\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.940168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.939989    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-metrics-client-ca\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.940168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.939999    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.940168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.940010    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-web-config\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.940168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.940018    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.940168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.940026    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-secret-alertmanager-main-tls\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.940168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.940035    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x9pp\" (UniqueName: \"kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-kube-api-access-4x9pp\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.940168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.940043    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8535c85-0583-4eae-bb67-f700361da016-config-out\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.940168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.940050    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8535c85-0583-4eae-bb67-f700361da016-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.940168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.940059    2567 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8535c85-0583-4eae-bb67-f700361da016-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:46.940168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:46.940066    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8535c85-0583-4eae-bb67-f700361da016-tls-assets\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:24:47.782658 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.782626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8535c85-0583-4eae-bb67-f700361da016","Type":"ContainerDied","Data":"e1b964ba4b5a8825fb9fcf7afe292df8602c540aa3e6ee67e814a26e7c6b2ff6"}
Sep 16 21:24:47.783019 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.782684    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.783019 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.782709    2567 scope.go:117] "RemoveContainer" containerID="309a948e31e9450643052d079f04168e6baeedd7cb7cc60776265c8d320f9b5b"
Sep 16 21:24:47.790747 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.790725    2567 scope.go:117] "RemoveContainer" containerID="e7efafa43252fa5e0df493ec39f1e5d7ff35d46f1ad2726e9b00155f0d27bc22"
Sep 16 21:24:47.799091 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.799071    2567 scope.go:117] "RemoveContainer" containerID="5c006d324c3f92ad3c5acd6d115bdf3200a3532b61605ad5c4840d0f6fc30813"
Sep 16 21:24:47.804223 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.804200    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 21:24:47.806349 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.806332    2567 scope.go:117] "RemoveContainer" containerID="51bfd8b0041640820cd93a6ffff3e4497a54961ea5c1107ab7f22fe6673c84d6"
Sep 16 21:24:47.808928 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.808909    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 21:24:47.812996 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.812972    2567 scope.go:117] "RemoveContainer" containerID="4132944ac85938718646abf056eb9baa0b9928b648fd7cf7e738b34df8db5b88"
Sep 16 21:24:47.819264 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.819249    2567 scope.go:117] "RemoveContainer" containerID="43aa79e3fe568b2dba717d860ad1545fbfa8799b8bb3efa56af2b9a9c86e64f6"
Sep 16 21:24:47.825802 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.825787    2567 scope.go:117] "RemoveContainer" containerID="57f0f05f24e799ff95cdbc21ccf086c192fdce780e42fef5b3822c12faabc4f0"
Sep 16 21:24:47.833467 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833437    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 21:24:47.833729 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833714    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy"
Sep 16 21:24:47.833806 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833732    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy"
Sep 16 21:24:47.833806 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833744    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="alertmanager"
Sep 16 21:24:47.833806 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833751    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="alertmanager"
Sep 16 21:24:47.833806 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833765    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="init-config-reloader"
Sep 16 21:24:47.833806 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833774    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="init-config-reloader"
Sep 16 21:24:47.833806 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833789    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy-metric"
Sep 16 21:24:47.833806 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833797    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy-metric"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833816    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="config-reloader"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833824    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="config-reloader"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833834    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy-web"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833844    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy-web"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833853    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="prom-label-proxy"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833862    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="prom-label-proxy"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833924    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833936    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="alertmanager"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833946    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="config-reloader"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833955    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="prom-label-proxy"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833963    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy-metric"
Sep 16 21:24:47.834132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.833973    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8535c85-0583-4eae-bb67-f700361da016" containerName="kube-rbac-proxy-web"
Sep 16 21:24:47.839054 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.839026    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.840955 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.840936    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 21:24:47.841181 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.841160    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nxgqm\""
Sep 16 21:24:47.841260 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.841202    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 21:24:47.841335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.841314    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 21:24:47.841419 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.841371    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 21:24:47.841516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.841470    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 21:24:47.841516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.841490    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 21:24:47.841619 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.841594    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 21:24:47.841619 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.841612    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 21:24:47.850607 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.850588    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 21:24:47.851575 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.851553    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 21:24:47.950024 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.949988    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-config-volume\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950024 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950028    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652abb68-3710-4f10-a7e0-7e49bef35826-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950219 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950064    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950219 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950109    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/652abb68-3710-4f10-a7e0-7e49bef35826-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950219 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950158    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/652abb68-3710-4f10-a7e0-7e49bef35826-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950219 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950187    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950260    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-web-config\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950283    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950341    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl4j9\" (UniqueName: \"kubernetes.io/projected/652abb68-3710-4f10-a7e0-7e49bef35826-kube-api-access-nl4j9\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950373    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950431    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950639 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950459    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/652abb68-3710-4f10-a7e0-7e49bef35826-config-out\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:47.950639 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:47.950488    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/652abb68-3710-4f10-a7e0-7e49bef35826-tls-assets\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051303 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051235    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051303 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051275    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/652abb68-3710-4f10-a7e0-7e49bef35826-config-out\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051303 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/652abb68-3710-4f10-a7e0-7e49bef35826-tls-assets\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051524 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051440    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-config-volume\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051524 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051475    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652abb68-3710-4f10-a7e0-7e49bef35826-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051524 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051511    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051679 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051535    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/652abb68-3710-4f10-a7e0-7e49bef35826-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051679 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051568    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/652abb68-3710-4f10-a7e0-7e49bef35826-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051679 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051597    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051679 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051649    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-web-config\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051679 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051918 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051708    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4j9\" (UniqueName: \"kubernetes.io/projected/652abb68-3710-4f10-a7e0-7e49bef35826-kube-api-access-nl4j9\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.051918 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.051736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.052588 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.052387    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652abb68-3710-4f10-a7e0-7e49bef35826-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.053214 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.053180    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/652abb68-3710-4f10-a7e0-7e49bef35826-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.054665 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.054468    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-config-volume\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.054665 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.054527    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.054665 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.054591    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/652abb68-3710-4f10-a7e0-7e49bef35826-tls-assets\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.054665 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.054620    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.054931 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.054705    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.054931 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.054723    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/652abb68-3710-4f10-a7e0-7e49bef35826-config-out\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.054931 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.054793    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/652abb68-3710-4f10-a7e0-7e49bef35826-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.055259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.055236    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-web-config\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.055709 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.055693    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.056489 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.056471    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/652abb68-3710-4f10-a7e0-7e49bef35826-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.061228 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.061208    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4j9\" (UniqueName: \"kubernetes.io/projected/652abb68-3710-4f10-a7e0-7e49bef35826-kube-api-access-nl4j9\") pod \"alertmanager-main-0\" (UID: \"652abb68-3710-4f10-a7e0-7e49bef35826\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.153672 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.153651    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 21:24:48.430690 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.430563    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 21:24:48.434386 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:24:48.434358    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652abb68_3710_4f10_a7e0_7e49bef35826.slice/crio-ef3f8eceacc04e19577844b1952731a7af39a0c9d8964c3ad6e8e95011052c70 WatchSource:0}: Error finding container ef3f8eceacc04e19577844b1952731a7af39a0c9d8964c3ad6e8e95011052c70: Status 404 returned error can't find the container with id ef3f8eceacc04e19577844b1952731a7af39a0c9d8964c3ad6e8e95011052c70
Sep 16 21:24:48.787976 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.787942    2567 generic.go:356] "Generic (PLEG): container finished" podID="652abb68-3710-4f10-a7e0-7e49bef35826" containerID="06a05639349bd5ef07c5b12f0aecb3a70ae4ae8a2eb95f140d7bab152a3c7a65" exitCode=0
Sep 16 21:24:48.788346 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.788024    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652abb68-3710-4f10-a7e0-7e49bef35826","Type":"ContainerDied","Data":"06a05639349bd5ef07c5b12f0aecb3a70ae4ae8a2eb95f140d7bab152a3c7a65"}
Sep 16 21:24:48.788346 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:48.788094    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652abb68-3710-4f10-a7e0-7e49bef35826","Type":"ContainerStarted","Data":"ef3f8eceacc04e19577844b1952731a7af39a0c9d8964c3ad6e8e95011052c70"}
Sep 16 21:24:49.107423 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:49.107379    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8535c85-0583-4eae-bb67-f700361da016" path="/var/lib/kubelet/pods/f8535c85-0583-4eae-bb67-f700361da016/volumes"
Sep 16 21:24:49.796265 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:49.796232    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652abb68-3710-4f10-a7e0-7e49bef35826","Type":"ContainerStarted","Data":"4c381e370491d7ebb26e7312c76ea725847151eff4e6aaf3f536705e9326f6fa"}
Sep 16 21:24:49.796655 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:49.796295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652abb68-3710-4f10-a7e0-7e49bef35826","Type":"ContainerStarted","Data":"ca64461f0536ee0a0271eab20f8ee1038e1b175625b6f2dcce1ddf743b390964"}
Sep 16 21:24:49.796655 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:49.796320    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652abb68-3710-4f10-a7e0-7e49bef35826","Type":"ContainerStarted","Data":"696ac997ce75f6fe3e6a9d299a377c777a3b779cab934a7bf1604f76c9b39c5c"}
Sep 16 21:24:49.796655 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:49.796346    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652abb68-3710-4f10-a7e0-7e49bef35826","Type":"ContainerStarted","Data":"6a5fb812949406b6d215b90df11d53e0f3fe80ac490c7990612e0a966d3e6b2d"}
Sep 16 21:24:49.796655 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:49.796370    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652abb68-3710-4f10-a7e0-7e49bef35826","Type":"ContainerStarted","Data":"d1e9cdb26f5267bc4126582db82e67a91f20f81611691342ed7736bc60540e99"}
Sep 16 21:24:49.796655 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:49.796413    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652abb68-3710-4f10-a7e0-7e49bef35826","Type":"ContainerStarted","Data":"029c37c643231937af4d2f6316d58aa3e92fffd35a1727e80fb355e7c850950e"}
Sep 16 21:24:49.820716 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:24:49.820670    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.801766653 podStartE2EDuration="2.820650318s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.883665ms" isStatefulPod=false podCreationTimestamp="2026-09-16 21:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:24:49.819165355 +0000 UTC m=+129.204848466" watchObservedRunningTime="2026-09-16 21:24:49.820650318 +0000 UTC m=+129.206333403"
Sep 16 21:25:56.761161 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.761127    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"]
Sep 16 21:25:56.764967 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.764949    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:25:56.767445 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.767421    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 21:25:56.767445 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.767438    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-shrwc\""
Sep 16 21:25:56.768028 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.768012    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 21:25:56.772468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.772425    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"]
Sep 16 21:25:56.858794 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.858763    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:25:56.858941 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.858800    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zz8w\" (UniqueName: \"kubernetes.io/projected/7ed34baa-de4d-43ab-b689-8d589149949a-kube-api-access-7zz8w\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:25:56.858941 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.858863    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:25:56.959841 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.959808    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:25:56.959991 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.959864    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:25:56.959991 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.959889    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zz8w\" (UniqueName: \"kubernetes.io/projected/7ed34baa-de4d-43ab-b689-8d589149949a-kube-api-access-7zz8w\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:25:56.960187 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.960166    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:25:56.960225 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.960195    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:25:56.967928 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:56.967899    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zz8w\" (UniqueName: \"kubernetes.io/projected/7ed34baa-de4d-43ab-b689-8d589149949a-kube-api-access-7zz8w\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:25:57.074856 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:57.074776    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:25:57.340612 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:57.340532    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"]
Sep 16 21:25:57.343412 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:25:57.343375    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed34baa_de4d_43ab_b689_8d589149949a.slice/crio-97aca7fb048468116a33e3cf24d7730ac411dd3e719ab141915f9cc3375dd26e WatchSource:0}: Error finding container 97aca7fb048468116a33e3cf24d7730ac411dd3e719ab141915f9cc3375dd26e: Status 404 returned error can't find the container with id 97aca7fb048468116a33e3cf24d7730ac411dd3e719ab141915f9cc3375dd26e
Sep 16 21:25:58.040623 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:25:58.040584    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k" event={"ID":"7ed34baa-de4d-43ab-b689-8d589149949a","Type":"ContainerStarted","Data":"97aca7fb048468116a33e3cf24d7730ac411dd3e719ab141915f9cc3375dd26e"}
Sep 16 21:26:04.065646 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:04.065618    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k" event={"ID":"7ed34baa-de4d-43ab-b689-8d589149949a","Type":"ContainerStarted","Data":"8d12d67a9394b7d2e4e67595509bda851fd1e809dc23c38c3b8e3a763f2f0d5c"}
Sep 16 21:26:05.071467 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:05.071432    2567 generic.go:356] "Generic (PLEG): container finished" podID="7ed34baa-de4d-43ab-b689-8d589149949a" containerID="8d12d67a9394b7d2e4e67595509bda851fd1e809dc23c38c3b8e3a763f2f0d5c" exitCode=0
Sep 16 21:26:05.071915 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:05.071505    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k" event={"ID":"7ed34baa-de4d-43ab-b689-8d589149949a","Type":"ContainerDied","Data":"8d12d67a9394b7d2e4e67595509bda851fd1e809dc23c38c3b8e3a763f2f0d5c"}
Sep 16 21:26:07.081601 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:07.081565    2567 generic.go:356] "Generic (PLEG): container finished" podID="7ed34baa-de4d-43ab-b689-8d589149949a" containerID="48950c207723546288d03b66f13ce5644735523da631fc7b758f0d0dc9010325" exitCode=0
Sep 16 21:26:07.081977 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:07.081611    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k" event={"ID":"7ed34baa-de4d-43ab-b689-8d589149949a","Type":"ContainerDied","Data":"48950c207723546288d03b66f13ce5644735523da631fc7b758f0d0dc9010325"}
Sep 16 21:26:14.112651 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:14.112615    2567 generic.go:356] "Generic (PLEG): container finished" podID="7ed34baa-de4d-43ab-b689-8d589149949a" containerID="0e3a5e7a480d669ac8a826b6a6b0d4381fbd506f369d665506490d461d39af3d" exitCode=0
Sep 16 21:26:14.113033 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:14.112691    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k" event={"ID":"7ed34baa-de4d-43ab-b689-8d589149949a","Type":"ContainerDied","Data":"0e3a5e7a480d669ac8a826b6a6b0d4381fbd506f369d665506490d461d39af3d"}
Sep 16 21:26:15.233307 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:15.233282    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:26:15.310056 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:15.310027    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-bundle\") pod \"7ed34baa-de4d-43ab-b689-8d589149949a\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") "
Sep 16 21:26:15.310216 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:15.310089    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-util\") pod \"7ed34baa-de4d-43ab-b689-8d589149949a\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") "
Sep 16 21:26:15.310216 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:15.310112    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zz8w\" (UniqueName: \"kubernetes.io/projected/7ed34baa-de4d-43ab-b689-8d589149949a-kube-api-access-7zz8w\") pod \"7ed34baa-de4d-43ab-b689-8d589149949a\" (UID: \"7ed34baa-de4d-43ab-b689-8d589149949a\") "
Sep 16 21:26:15.310742 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:15.310717    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-bundle" (OuterVolumeSpecName: "bundle") pod "7ed34baa-de4d-43ab-b689-8d589149949a" (UID: "7ed34baa-de4d-43ab-b689-8d589149949a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:26:15.312259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:15.312239    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed34baa-de4d-43ab-b689-8d589149949a-kube-api-access-7zz8w" (OuterVolumeSpecName: "kube-api-access-7zz8w") pod "7ed34baa-de4d-43ab-b689-8d589149949a" (UID: "7ed34baa-de4d-43ab-b689-8d589149949a"). InnerVolumeSpecName "kube-api-access-7zz8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:26:15.314323 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:15.314304    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-util" (OuterVolumeSpecName: "util") pod "7ed34baa-de4d-43ab-b689-8d589149949a" (UID: "7ed34baa-de4d-43ab-b689-8d589149949a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:26:15.410997 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:15.410912    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-util\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:26:15.410997 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:15.410942    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zz8w\" (UniqueName: \"kubernetes.io/projected/7ed34baa-de4d-43ab-b689-8d589149949a-kube-api-access-7zz8w\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:26:15.410997 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:15.410952    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ed34baa-de4d-43ab-b689-8d589149949a-bundle\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:26:16.121157 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:16.121125    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k"
Sep 16 21:26:16.121157 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:16.121146    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404dn4k" event={"ID":"7ed34baa-de4d-43ab-b689-8d589149949a","Type":"ContainerDied","Data":"97aca7fb048468116a33e3cf24d7730ac411dd3e719ab141915f9cc3375dd26e"}
Sep 16 21:26:16.121357 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:16.121190    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97aca7fb048468116a33e3cf24d7730ac411dd3e719ab141915f9cc3375dd26e"
Sep 16 21:26:19.929826 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.929745    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs"]
Sep 16 21:26:19.930193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.930055    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed34baa-de4d-43ab-b689-8d589149949a" containerName="extract"
Sep 16 21:26:19.930193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.930067    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed34baa-de4d-43ab-b689-8d589149949a" containerName="extract"
Sep 16 21:26:19.930193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.930080    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed34baa-de4d-43ab-b689-8d589149949a" containerName="pull"
Sep 16 21:26:19.930193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.930085    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed34baa-de4d-43ab-b689-8d589149949a" containerName="pull"
Sep 16 21:26:19.930193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.930095    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed34baa-de4d-43ab-b689-8d589149949a" containerName="util"
Sep 16 21:26:19.930193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.930101    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed34baa-de4d-43ab-b689-8d589149949a" containerName="util"
Sep 16 21:26:19.930193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.930158    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed34baa-de4d-43ab-b689-8d589149949a" containerName="extract"
Sep 16 21:26:19.932237 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.932221    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs"
Sep 16 21:26:19.934292 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.934268    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:26:19.934459 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.934412    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 21:26:19.934647 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.934629    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-f4hgq\""
Sep 16 21:26:19.944556 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:19.944536    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs"]
Sep 16 21:26:20.041949 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:20.041922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bd8afe0-519d-468a-830f-586975556e81-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2f8gs\" (UID: \"3bd8afe0-519d-468a-830f-586975556e81\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs"
Sep 16 21:26:20.042060 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:20.041959    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258w4\" (UniqueName: \"kubernetes.io/projected/3bd8afe0-519d-468a-830f-586975556e81-kube-api-access-258w4\") pod \"cert-manager-operator-controller-manager-55c444bf86-2f8gs\" (UID: \"3bd8afe0-519d-468a-830f-586975556e81\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs"
Sep 16 21:26:20.142874 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:20.142853    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bd8afe0-519d-468a-830f-586975556e81-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2f8gs\" (UID: \"3bd8afe0-519d-468a-830f-586975556e81\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs"
Sep 16 21:26:20.142997 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:20.142882    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258w4\" (UniqueName: \"kubernetes.io/projected/3bd8afe0-519d-468a-830f-586975556e81-kube-api-access-258w4\") pod \"cert-manager-operator-controller-manager-55c444bf86-2f8gs\" (UID: \"3bd8afe0-519d-468a-830f-586975556e81\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs"
Sep 16 21:26:20.143231 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:20.143210    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bd8afe0-519d-468a-830f-586975556e81-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2f8gs\" (UID: \"3bd8afe0-519d-468a-830f-586975556e81\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs"
Sep 16 21:26:20.151000 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:20.150978    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-258w4\" (UniqueName: \"kubernetes.io/projected/3bd8afe0-519d-468a-830f-586975556e81-kube-api-access-258w4\") pod \"cert-manager-operator-controller-manager-55c444bf86-2f8gs\" (UID: \"3bd8afe0-519d-468a-830f-586975556e81\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs"
Sep 16 21:26:20.240771 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:20.240748    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs"
Sep 16 21:26:20.520523 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:20.520407    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs"]
Sep 16 21:26:20.525122 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:26:20.525094    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd8afe0_519d_468a_830f_586975556e81.slice/crio-025e1408b7094675690d611cc7e800d2378b0492ace5f14b567acd799181331f WatchSource:0}: Error finding container 025e1408b7094675690d611cc7e800d2378b0492ace5f14b567acd799181331f: Status 404 returned error can't find the container with id 025e1408b7094675690d611cc7e800d2378b0492ace5f14b567acd799181331f
Sep 16 21:26:21.140307 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:21.140275    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs" event={"ID":"3bd8afe0-519d-468a-830f-586975556e81","Type":"ContainerStarted","Data":"025e1408b7094675690d611cc7e800d2378b0492ace5f14b567acd799181331f"}
Sep 16 21:26:23.148958 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:23.148919    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs" event={"ID":"3bd8afe0-519d-468a-830f-586975556e81","Type":"ContainerStarted","Data":"9c1ade3a287e84d6368dafaba8c9124e8f7664cde2f32a7d7954c1bef11c124b"}
Sep 16 21:26:23.168834 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:23.168780    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2f8gs" podStartSLOduration=2.533929994 podStartE2EDuration="4.168760163s" totalImagesPullingTime="1.634830169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:26:23.166371777 +0000 UTC m=+222.552054861" watchObservedRunningTime="2026-09-16 21:26:23.168760163 +0000 UTC m=+222.554443248"
Sep 16 21:26:24.276957 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.276922    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"]
Sep 16 21:26:24.279346 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.279330    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:24.281413 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.281372    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 21:26:24.281534 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.281442    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 21:26:24.281534 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.281459    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-shrwc\""
Sep 16 21:26:24.291625 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.291605    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"]
Sep 16 21:26:24.333124 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.333098    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6qmkk"]
Sep 16 21:26:24.335217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.335201    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk"
Sep 16 21:26:24.337301 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.337273    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 21:26:24.337391 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.337305    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 21:26:24.337391 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.337362    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jwj7z\""
Sep 16 21:26:24.344940 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.344920    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6qmkk"]
Sep 16 21:26:24.374848 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.374810    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:24.374956 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.374869    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:24.374956 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.374898    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mv64\" (UniqueName: \"kubernetes.io/projected/25217130-5b59-403d-bafa-0236850a3c8b-kube-api-access-2mv64\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:24.476024 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.475994    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:24.476024 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.476034    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq9gt\" (UniqueName: \"kubernetes.io/projected/7747cf3f-7846-422c-a4f9-8c2776cfbefd-kube-api-access-vq9gt\") pod \"cert-manager-webhook-6486ffd997-6qmkk\" (UID: \"7747cf3f-7846-422c-a4f9-8c2776cfbefd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk"
Sep 16 21:26:24.476259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.476061    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7747cf3f-7846-422c-a4f9-8c2776cfbefd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6qmkk\" (UID: \"7747cf3f-7846-422c-a4f9-8c2776cfbefd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk"
Sep 16 21:26:24.476259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.476081    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:24.476259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.476125    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mv64\" (UniqueName: \"kubernetes.io/projected/25217130-5b59-403d-bafa-0236850a3c8b-kube-api-access-2mv64\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:24.476486 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.476468    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:24.476544 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.476491    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:24.486781 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.486758    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mv64\" (UniqueName: \"kubernetes.io/projected/25217130-5b59-403d-bafa-0236850a3c8b-kube-api-access-2mv64\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:24.577416 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.577309    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq9gt\" (UniqueName: \"kubernetes.io/projected/7747cf3f-7846-422c-a4f9-8c2776cfbefd-kube-api-access-vq9gt\") pod \"cert-manager-webhook-6486ffd997-6qmkk\" (UID: \"7747cf3f-7846-422c-a4f9-8c2776cfbefd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk"
Sep 16 21:26:24.577573 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.577442    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7747cf3f-7846-422c-a4f9-8c2776cfbefd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6qmkk\" (UID: \"7747cf3f-7846-422c-a4f9-8c2776cfbefd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk"
Sep 16 21:26:24.585016 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.584988    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7747cf3f-7846-422c-a4f9-8c2776cfbefd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6qmkk\" (UID: \"7747cf3f-7846-422c-a4f9-8c2776cfbefd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk"
Sep 16 21:26:24.585107 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.585087    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq9gt\" (UniqueName: \"kubernetes.io/projected/7747cf3f-7846-422c-a4f9-8c2776cfbefd-kube-api-access-vq9gt\") pod \"cert-manager-webhook-6486ffd997-6qmkk\" (UID: \"7747cf3f-7846-422c-a4f9-8c2776cfbefd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk"
Sep 16 21:26:24.587910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.587892    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:24.662414 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.662366    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk"
Sep 16 21:26:24.857849 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.857820    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"]
Sep 16 21:26:24.860283 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:26:24.860252    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25217130_5b59_403d_bafa_0236850a3c8b.slice/crio-c371c1e02805e9a19ba599d1617a7d7032bd1f833976180298a04320f042f433 WatchSource:0}: Error finding container c371c1e02805e9a19ba599d1617a7d7032bd1f833976180298a04320f042f433: Status 404 returned error can't find the container with id c371c1e02805e9a19ba599d1617a7d7032bd1f833976180298a04320f042f433
Sep 16 21:26:24.931947 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:24.931914    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6qmkk"]
Sep 16 21:26:24.935821 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:26:24.935797    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7747cf3f_7846_422c_a4f9_8c2776cfbefd.slice/crio-6cce3753573be0e5de3ea732926846ba2982c942c790a57e8eaeb7676c649601 WatchSource:0}: Error finding container 6cce3753573be0e5de3ea732926846ba2982c942c790a57e8eaeb7676c649601: Status 404 returned error can't find the container with id 6cce3753573be0e5de3ea732926846ba2982c942c790a57e8eaeb7676c649601
Sep 16 21:26:25.157602 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:25.157520    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk" event={"ID":"7747cf3f-7846-422c-a4f9-8c2776cfbefd","Type":"ContainerStarted","Data":"6cce3753573be0e5de3ea732926846ba2982c942c790a57e8eaeb7676c649601"}
Sep 16 21:26:25.159007 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:25.158981    2567 generic.go:356] "Generic (PLEG): container finished" podID="25217130-5b59-403d-bafa-0236850a3c8b" containerID="fdef879c5b870201a324bddb5822015f5d49f9085bd2a321eee8c0a6e36c07d5" exitCode=0
Sep 16 21:26:25.159125 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:25.159045    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4" event={"ID":"25217130-5b59-403d-bafa-0236850a3c8b","Type":"ContainerDied","Data":"fdef879c5b870201a324bddb5822015f5d49f9085bd2a321eee8c0a6e36c07d5"}
Sep 16 21:26:25.159125 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:25.159091    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4" event={"ID":"25217130-5b59-403d-bafa-0236850a3c8b","Type":"ContainerStarted","Data":"c371c1e02805e9a19ba599d1617a7d7032bd1f833976180298a04320f042f433"}
Sep 16 21:26:27.413912 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:27.413878    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7l47h"]
Sep 16 21:26:27.416513 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:27.416495    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7l47h"
Sep 16 21:26:27.418612 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:27.418587    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gwr6x\""
Sep 16 21:26:27.425745 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:27.425722    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7l47h"]
Sep 16 21:26:27.502855 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:27.502828    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75087730-3052-4dda-a2bf-e6d05177a219-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7l47h\" (UID: \"75087730-3052-4dda-a2bf-e6d05177a219\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7l47h"
Sep 16 21:26:27.503012 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:27.502910    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxptr\" (UniqueName: \"kubernetes.io/projected/75087730-3052-4dda-a2bf-e6d05177a219-kube-api-access-zxptr\") pod \"cert-manager-cainjector-559cdd8c87-7l47h\" (UID: \"75087730-3052-4dda-a2bf-e6d05177a219\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7l47h"
Sep 16 21:26:27.604312 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:27.604282    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxptr\" (UniqueName: \"kubernetes.io/projected/75087730-3052-4dda-a2bf-e6d05177a219-kube-api-access-zxptr\") pod \"cert-manager-cainjector-559cdd8c87-7l47h\" (UID: \"75087730-3052-4dda-a2bf-e6d05177a219\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7l47h"
Sep 16 21:26:27.604486 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:27.604340    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75087730-3052-4dda-a2bf-e6d05177a219-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7l47h\" (UID: \"75087730-3052-4dda-a2bf-e6d05177a219\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7l47h"
Sep 16 21:26:27.613805 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:27.613782    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75087730-3052-4dda-a2bf-e6d05177a219-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7l47h\" (UID: \"75087730-3052-4dda-a2bf-e6d05177a219\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7l47h"
Sep 16 21:26:27.613950 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:27.613928    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxptr\" (UniqueName: \"kubernetes.io/projected/75087730-3052-4dda-a2bf-e6d05177a219-kube-api-access-zxptr\") pod \"cert-manager-cainjector-559cdd8c87-7l47h\" (UID: \"75087730-3052-4dda-a2bf-e6d05177a219\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7l47h"
Sep 16 21:26:27.726748 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:27.726721    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7l47h"
Sep 16 21:26:28.467501 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:28.467469    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7l47h"]
Sep 16 21:26:28.470873 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:26:28.470842    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75087730_3052_4dda_a2bf_e6d05177a219.slice/crio-95276ffb4692794f46f2fae5360f6b068e1108c1633b7971c9bbf6e14c396a7c WatchSource:0}: Error finding container 95276ffb4692794f46f2fae5360f6b068e1108c1633b7971c9bbf6e14c396a7c: Status 404 returned error can't find the container with id 95276ffb4692794f46f2fae5360f6b068e1108c1633b7971c9bbf6e14c396a7c
Sep 16 21:26:29.180389 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:29.180353    2567 generic.go:356] "Generic (PLEG): container finished" podID="25217130-5b59-403d-bafa-0236850a3c8b" containerID="68b7cc16dcaa805451d7545877fae42c83e1cd80a498fcf65643e1e0a3b32b93" exitCode=0
Sep 16 21:26:29.180589 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:29.180446    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4" event={"ID":"25217130-5b59-403d-bafa-0236850a3c8b","Type":"ContainerDied","Data":"68b7cc16dcaa805451d7545877fae42c83e1cd80a498fcf65643e1e0a3b32b93"}
Sep 16 21:26:29.182118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:29.182084    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7l47h" event={"ID":"75087730-3052-4dda-a2bf-e6d05177a219","Type":"ContainerStarted","Data":"248b03cb6e0248872c587b926124cd1e6a27b3aa461516acf76ddeeb55a6c02c"}
Sep 16 21:26:29.182228 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:29.182123    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7l47h" event={"ID":"75087730-3052-4dda-a2bf-e6d05177a219","Type":"ContainerStarted","Data":"95276ffb4692794f46f2fae5360f6b068e1108c1633b7971c9bbf6e14c396a7c"}
Sep 16 21:26:29.183552 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:29.183536    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk" event={"ID":"7747cf3f-7846-422c-a4f9-8c2776cfbefd","Type":"ContainerStarted","Data":"84ca6ca9dba934e299659b83874d86035d5b66eae026ba54144e47d398f7fdf3"}
Sep 16 21:26:29.183679 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:29.183667    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk"
Sep 16 21:26:29.211219 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:29.211181    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7l47h" podStartSLOduration=2.211168503 podStartE2EDuration="2.211168503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:26:29.209464615 +0000 UTC m=+228.595147699" watchObservedRunningTime="2026-09-16 21:26:29.211168503 +0000 UTC m=+228.596851588"
Sep 16 21:26:29.223874 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:29.223822    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk" podStartSLOduration=1.9565931559999998 podStartE2EDuration="5.223805213s" totalImagesPullingTime="3.267212057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:26:29.223331667 +0000 UTC m=+228.609014767" watchObservedRunningTime="2026-09-16 21:26:29.223805213 +0000 UTC m=+228.609488298"
Sep 16 21:26:30.190921 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:30.190883    2567 generic.go:356] "Generic (PLEG): container finished" podID="25217130-5b59-403d-bafa-0236850a3c8b" containerID="32410e3c0ba139b30bb3773eb9368b9ca7e89be43af7aab944e18bda69518853" exitCode=0
Sep 16 21:26:30.191296 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:30.190973    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4" event={"ID":"25217130-5b59-403d-bafa-0236850a3c8b","Type":"ContainerDied","Data":"32410e3c0ba139b30bb3773eb9368b9ca7e89be43af7aab944e18bda69518853"}
Sep 16 21:26:31.316356 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:31.316332    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:31.432338 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:31.432309    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-bundle\") pod \"25217130-5b59-403d-bafa-0236850a3c8b\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") "
Sep 16 21:26:31.432489 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:31.432349    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-util\") pod \"25217130-5b59-403d-bafa-0236850a3c8b\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") "
Sep 16 21:26:31.432489 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:31.432420    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mv64\" (UniqueName: \"kubernetes.io/projected/25217130-5b59-403d-bafa-0236850a3c8b-kube-api-access-2mv64\") pod \"25217130-5b59-403d-bafa-0236850a3c8b\" (UID: \"25217130-5b59-403d-bafa-0236850a3c8b\") "
Sep 16 21:26:31.432678 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:31.432656    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-bundle" (OuterVolumeSpecName: "bundle") pod "25217130-5b59-403d-bafa-0236850a3c8b" (UID: "25217130-5b59-403d-bafa-0236850a3c8b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:26:31.434345 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:31.434316    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25217130-5b59-403d-bafa-0236850a3c8b-kube-api-access-2mv64" (OuterVolumeSpecName: "kube-api-access-2mv64") pod "25217130-5b59-403d-bafa-0236850a3c8b" (UID: "25217130-5b59-403d-bafa-0236850a3c8b"). InnerVolumeSpecName "kube-api-access-2mv64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:26:31.436911 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:31.436886    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-util" (OuterVolumeSpecName: "util") pod "25217130-5b59-403d-bafa-0236850a3c8b" (UID: "25217130-5b59-403d-bafa-0236850a3c8b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:26:31.533563 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:31.533526    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mv64\" (UniqueName: \"kubernetes.io/projected/25217130-5b59-403d-bafa-0236850a3c8b-kube-api-access-2mv64\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:26:31.533563 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:31.533560    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-bundle\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:26:31.533563 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:31.533570    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25217130-5b59-403d-bafa-0236850a3c8b-util\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:26:32.201372 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:32.201341    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4"
Sep 16 21:26:32.201574 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:32.201348    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjb4m4" event={"ID":"25217130-5b59-403d-bafa-0236850a3c8b","Type":"ContainerDied","Data":"c371c1e02805e9a19ba599d1617a7d7032bd1f833976180298a04320f042f433"}
Sep 16 21:26:32.201574 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:32.201444    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c371c1e02805e9a19ba599d1617a7d7032bd1f833976180298a04320f042f433"
Sep 16 21:26:35.193419 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:35.193371    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6qmkk"
Sep 16 21:26:38.183682 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.183647    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f"]
Sep 16 21:26:38.184033 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.183934    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25217130-5b59-403d-bafa-0236850a3c8b" containerName="pull"
Sep 16 21:26:38.184033 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.183945    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="25217130-5b59-403d-bafa-0236850a3c8b" containerName="pull"
Sep 16 21:26:38.184033 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.183961    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25217130-5b59-403d-bafa-0236850a3c8b" containerName="util"
Sep 16 21:26:38.184033 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.183966    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="25217130-5b59-403d-bafa-0236850a3c8b" containerName="util"
Sep 16 21:26:38.184033 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.183976    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25217130-5b59-403d-bafa-0236850a3c8b" containerName="extract"
Sep 16 21:26:38.184033 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.183983    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="25217130-5b59-403d-bafa-0236850a3c8b" containerName="extract"
Sep 16 21:26:38.184033 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.184026    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="25217130-5b59-403d-bafa-0236850a3c8b" containerName="extract"
Sep 16 21:26:38.188185 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.188164    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f"
Sep 16 21:26:38.190470 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.190444    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:26:38.191049 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.191025    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fvk9l\""
Sep 16 21:26:38.191168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.191027    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 21:26:38.195143 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.194886    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f"]
Sep 16 21:26:38.281226 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.281195    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4bf756-e00e-45af-9659-9233f59f9ee6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wn77f\" (UID: \"cb4bf756-e00e-45af-9659-9233f59f9ee6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f"
Sep 16 21:26:38.281358 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.281257    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fd2q\" (UniqueName: \"kubernetes.io/projected/cb4bf756-e00e-45af-9659-9233f59f9ee6-kube-api-access-2fd2q\") pod \"openshift-lws-operator-fd8ccff4c-wn77f\" (UID: \"cb4bf756-e00e-45af-9659-9233f59f9ee6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f"
Sep 16 21:26:38.381667 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.381640    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fd2q\" (UniqueName: \"kubernetes.io/projected/cb4bf756-e00e-45af-9659-9233f59f9ee6-kube-api-access-2fd2q\") pod \"openshift-lws-operator-fd8ccff4c-wn77f\" (UID: \"cb4bf756-e00e-45af-9659-9233f59f9ee6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f"
Sep 16 21:26:38.381790 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.381698    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4bf756-e00e-45af-9659-9233f59f9ee6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wn77f\" (UID: \"cb4bf756-e00e-45af-9659-9233f59f9ee6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f"
Sep 16 21:26:38.382062 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.382041    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4bf756-e00e-45af-9659-9233f59f9ee6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wn77f\" (UID: \"cb4bf756-e00e-45af-9659-9233f59f9ee6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f"
Sep 16 21:26:38.389247 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.389229    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fd2q\" (UniqueName: \"kubernetes.io/projected/cb4bf756-e00e-45af-9659-9233f59f9ee6-kube-api-access-2fd2q\") pod \"openshift-lws-operator-fd8ccff4c-wn77f\" (UID: \"cb4bf756-e00e-45af-9659-9233f59f9ee6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f"
Sep 16 21:26:38.497264 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.497228    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f"
Sep 16 21:26:38.764942 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:38.764914    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f"]
Sep 16 21:26:38.767739 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:26:38.767715    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb4bf756_e00e_45af_9659_9233f59f9ee6.slice/crio-ed9934f2b922e8dceb8093dfb8fe3c403a8101d2a842ed62e24bea00336af367 WatchSource:0}: Error finding container ed9934f2b922e8dceb8093dfb8fe3c403a8101d2a842ed62e24bea00336af367: Status 404 returned error can't find the container with id ed9934f2b922e8dceb8093dfb8fe3c403a8101d2a842ed62e24bea00336af367
Sep 16 21:26:39.231245 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:39.231218    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f" event={"ID":"cb4bf756-e00e-45af-9659-9233f59f9ee6","Type":"ContainerStarted","Data":"ed9934f2b922e8dceb8093dfb8fe3c403a8101d2a842ed62e24bea00336af367"}
Sep 16 21:26:41.242593 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:41.242555    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f" event={"ID":"cb4bf756-e00e-45af-9659-9233f59f9ee6","Type":"ContainerStarted","Data":"d15ae04e86896eea56137633c650f7cea06f9f30605ffc57f1bfa376a76401e3"}
Sep 16 21:26:41.264185 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:41.264130    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wn77f" podStartSLOduration=1.2462911110000001 podStartE2EDuration="3.264112294s" totalImagesPullingTime="2.017821183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:26:41.26238367 +0000 UTC m=+240.648066766" watchObservedRunningTime="2026-09-16 21:26:41.264112294 +0000 UTC m=+240.649795380"
Sep 16 21:26:43.640836 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.640806    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"]
Sep 16 21:26:43.656664 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.656640    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"]
Sep 16 21:26:43.656805 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.656736    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:43.659270 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.659242    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 21:26:43.659270 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.659242    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 21:26:43.659993 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.659975    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-shrwc\""
Sep 16 21:26:43.825063 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.825025    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5hf\" (UniqueName: \"kubernetes.io/projected/502a73ca-e5f8-4385-87cc-823a698e030d-kube-api-access-sw5hf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:43.825215 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.825137    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:43.825215 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.825181    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:43.926115 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.926052    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5hf\" (UniqueName: \"kubernetes.io/projected/502a73ca-e5f8-4385-87cc-823a698e030d-kube-api-access-sw5hf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:43.926222 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.926115    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:43.926222 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.926149    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:43.926592 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.926573    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:43.926643 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.926593    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:43.947087 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.947065    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5hf\" (UniqueName: \"kubernetes.io/projected/502a73ca-e5f8-4385-87cc-823a698e030d-kube-api-access-sw5hf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:43.966546 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:43.966527    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:44.252199 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:44.252061    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"]
Sep 16 21:26:44.254376 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:26:44.254345    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502a73ca_e5f8_4385_87cc_823a698e030d.slice/crio-02acaa311c98c0b87198159db8359a8fa947d12e59657616169ca38d50a87262 WatchSource:0}: Error finding container 02acaa311c98c0b87198159db8359a8fa947d12e59657616169ca38d50a87262: Status 404 returned error can't find the container with id 02acaa311c98c0b87198159db8359a8fa947d12e59657616169ca38d50a87262
Sep 16 21:26:45.261558 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:45.261473    2567 generic.go:356] "Generic (PLEG): container finished" podID="502a73ca-e5f8-4385-87cc-823a698e030d" containerID="65e916bb096a0620a75a9ba4a56fea14b54161f353f6b4da8a2bb4eb38b484f7" exitCode=0
Sep 16 21:26:45.261558 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:45.261530    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq" event={"ID":"502a73ca-e5f8-4385-87cc-823a698e030d","Type":"ContainerDied","Data":"65e916bb096a0620a75a9ba4a56fea14b54161f353f6b4da8a2bb4eb38b484f7"}
Sep 16 21:26:45.262094 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:45.261572    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq" event={"ID":"502a73ca-e5f8-4385-87cc-823a698e030d","Type":"ContainerStarted","Data":"02acaa311c98c0b87198159db8359a8fa947d12e59657616169ca38d50a87262"}
Sep 16 21:26:46.267846 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:46.267760    2567 generic.go:356] "Generic (PLEG): container finished" podID="502a73ca-e5f8-4385-87cc-823a698e030d" containerID="ccb2b3f2e13f8abd228aef64650e4bbdd2753cebd2a2a9a6e9eb5d4ea293485b" exitCode=0
Sep 16 21:26:46.267846 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:46.267825    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq" event={"ID":"502a73ca-e5f8-4385-87cc-823a698e030d","Type":"ContainerDied","Data":"ccb2b3f2e13f8abd228aef64650e4bbdd2753cebd2a2a9a6e9eb5d4ea293485b"}
Sep 16 21:26:47.273900 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:47.273859    2567 generic.go:356] "Generic (PLEG): container finished" podID="502a73ca-e5f8-4385-87cc-823a698e030d" containerID="1434dd860ddd0c2240c431bbec5ad716adbff3cd8886305b1419306e3041ed01" exitCode=0
Sep 16 21:26:47.274280 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:47.273944    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq" event={"ID":"502a73ca-e5f8-4385-87cc-823a698e030d","Type":"ContainerDied","Data":"1434dd860ddd0c2240c431bbec5ad716adbff3cd8886305b1419306e3041ed01"}
Sep 16 21:26:48.394567 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:48.394540    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:48.564332 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:48.564246    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw5hf\" (UniqueName: \"kubernetes.io/projected/502a73ca-e5f8-4385-87cc-823a698e030d-kube-api-access-sw5hf\") pod \"502a73ca-e5f8-4385-87cc-823a698e030d\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") "
Sep 16 21:26:48.564332 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:48.564321    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-util\") pod \"502a73ca-e5f8-4385-87cc-823a698e030d\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") "
Sep 16 21:26:48.564584 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:48.564352    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-bundle\") pod \"502a73ca-e5f8-4385-87cc-823a698e030d\" (UID: \"502a73ca-e5f8-4385-87cc-823a698e030d\") "
Sep 16 21:26:48.565053 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:48.565022    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-bundle" (OuterVolumeSpecName: "bundle") pod "502a73ca-e5f8-4385-87cc-823a698e030d" (UID: "502a73ca-e5f8-4385-87cc-823a698e030d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:26:48.566444 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:48.566412    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502a73ca-e5f8-4385-87cc-823a698e030d-kube-api-access-sw5hf" (OuterVolumeSpecName: "kube-api-access-sw5hf") pod "502a73ca-e5f8-4385-87cc-823a698e030d" (UID: "502a73ca-e5f8-4385-87cc-823a698e030d"). InnerVolumeSpecName "kube-api-access-sw5hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:26:48.569807 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:48.569786    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-util" (OuterVolumeSpecName: "util") pod "502a73ca-e5f8-4385-87cc-823a698e030d" (UID: "502a73ca-e5f8-4385-87cc-823a698e030d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:26:48.665077 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:48.665053    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-util\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:26:48.665077 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:48.665073    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/502a73ca-e5f8-4385-87cc-823a698e030d-bundle\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:26:48.665226 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:48.665081    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5hf\" (UniqueName: \"kubernetes.io/projected/502a73ca-e5f8-4385-87cc-823a698e030d-kube-api-access-sw5hf\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:26:49.284323 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:49.284294    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq"
Sep 16 21:26:49.284504 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:49.284320    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwsqkq" event={"ID":"502a73ca-e5f8-4385-87cc-823a698e030d","Type":"ContainerDied","Data":"02acaa311c98c0b87198159db8359a8fa947d12e59657616169ca38d50a87262"}
Sep 16 21:26:49.284504 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:49.284368    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02acaa311c98c0b87198159db8359a8fa947d12e59657616169ca38d50a87262"
Sep 16 21:26:59.623544 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.623505    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"]
Sep 16 21:26:59.624102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.623910    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="502a73ca-e5f8-4385-87cc-823a698e030d" containerName="util"
Sep 16 21:26:59.624102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.623929    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="502a73ca-e5f8-4385-87cc-823a698e030d" containerName="util"
Sep 16 21:26:59.624102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.623948    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="502a73ca-e5f8-4385-87cc-823a698e030d" containerName="extract"
Sep 16 21:26:59.624102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.623957    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="502a73ca-e5f8-4385-87cc-823a698e030d" containerName="extract"
Sep 16 21:26:59.624102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.623979    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="502a73ca-e5f8-4385-87cc-823a698e030d" containerName="pull"
Sep 16 21:26:59.624102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.623989    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="502a73ca-e5f8-4385-87cc-823a698e030d" containerName="pull"
Sep 16 21:26:59.624102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.624068    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="502a73ca-e5f8-4385-87cc-823a698e030d" containerName="extract"
Sep 16 21:26:59.627026 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.627006    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:26:59.629536 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.629511    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 21:26:59.630104 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.630090    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 21:26:59.630166 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.630099    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-shrwc\""
Sep 16 21:26:59.636924 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.636903    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"]
Sep 16 21:26:59.647099 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.647080    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:26:59.647182 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.647120    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:26:59.647182 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.647156    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4scsn\" (UniqueName: \"kubernetes.io/projected/09d49cf5-96c1-4308-b7da-7ff47d520ccd-kube-api-access-4scsn\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:26:59.747865 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.747840    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:26:59.747973 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.747879    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4scsn\" (UniqueName: \"kubernetes.io/projected/09d49cf5-96c1-4308-b7da-7ff47d520ccd-kube-api-access-4scsn\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:26:59.747973 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.747914    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:26:59.748234 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.748214    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:26:59.748293 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.748237    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:26:59.755927 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.755907    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4scsn\" (UniqueName: \"kubernetes.io/projected/09d49cf5-96c1-4308-b7da-7ff47d520ccd-kube-api-access-4scsn\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:26:59.936483 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:26:59.936384    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:27:00.215541 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:00.212251    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"]
Sep 16 21:27:00.219159 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:27:00.219123    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d49cf5_96c1_4308_b7da_7ff47d520ccd.slice/crio-df069ac2c01f58245e2ffba6a1b3db824dc57c003e025df943e68075ee7eec7c WatchSource:0}: Error finding container df069ac2c01f58245e2ffba6a1b3db824dc57c003e025df943e68075ee7eec7c: Status 404 returned error can't find the container with id df069ac2c01f58245e2ffba6a1b3db824dc57c003e025df943e68075ee7eec7c
Sep 16 21:27:00.332374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:00.332347    2567 generic.go:356] "Generic (PLEG): container finished" podID="09d49cf5-96c1-4308-b7da-7ff47d520ccd" containerID="e05e6c589fca6da4dc57a56364b3f8a1d5c7d1c76f636b588bab352c5a7dba57" exitCode=0
Sep 16 21:27:00.332526 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:00.332433    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl" event={"ID":"09d49cf5-96c1-4308-b7da-7ff47d520ccd","Type":"ContainerDied","Data":"e05e6c589fca6da4dc57a56364b3f8a1d5c7d1c76f636b588bab352c5a7dba57"}
Sep 16 21:27:00.332526 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:00.332494    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl" event={"ID":"09d49cf5-96c1-4308-b7da-7ff47d520ccd","Type":"ContainerStarted","Data":"df069ac2c01f58245e2ffba6a1b3db824dc57c003e025df943e68075ee7eec7c"}
Sep 16 21:27:01.008562 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.008532    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"]
Sep 16 21:27:01.011169 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.011122    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.013505 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.013429    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 21:27:01.013631 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.013615    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 21:27:01.013789 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.013771    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 21:27:01.014045 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.014028    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lk5q9\""
Sep 16 21:27:01.014177 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.014159    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 21:27:01.029898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.029871    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"]
Sep 16 21:27:01.057532 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.057499    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/611804fc-3070-49eb-b152-b2459ec900cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.057633 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.057551    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/611804fc-3070-49eb-b152-b2459ec900cb-tmp\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.057633 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.057612    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqpw\" (UniqueName: \"kubernetes.io/projected/611804fc-3070-49eb-b152-b2459ec900cb-kube-api-access-frqpw\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.057633 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.057629    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/611804fc-3070-49eb-b152-b2459ec900cb-manifests\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.057732 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.057658    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/611804fc-3070-49eb-b152-b2459ec900cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.158683 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.158656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqpw\" (UniqueName: \"kubernetes.io/projected/611804fc-3070-49eb-b152-b2459ec900cb-kube-api-access-frqpw\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.158811 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.158691    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/611804fc-3070-49eb-b152-b2459ec900cb-manifests\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.158811 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.158719    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/611804fc-3070-49eb-b152-b2459ec900cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.158811 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.158764    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/611804fc-3070-49eb-b152-b2459ec900cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.158811 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.158807    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/611804fc-3070-49eb-b152-b2459ec900cb-tmp\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.159209 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.159171    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/611804fc-3070-49eb-b152-b2459ec900cb-manifests\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.161009 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.160987    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/611804fc-3070-49eb-b152-b2459ec900cb-tmp\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.161425 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.161387    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/611804fc-3070-49eb-b152-b2459ec900cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.161832 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.161810    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/611804fc-3070-49eb-b152-b2459ec900cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.172554 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.172529    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqpw\" (UniqueName: \"kubernetes.io/projected/611804fc-3070-49eb-b152-b2459ec900cb-kube-api-access-frqpw\") pod \"opendatahub-operator-controller-manager-df9d657c6-dpdnf\" (UID: \"611804fc-3070-49eb-b152-b2459ec900cb\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.322806 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.322724    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:01.339997 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.339969    2567 generic.go:356] "Generic (PLEG): container finished" podID="09d49cf5-96c1-4308-b7da-7ff47d520ccd" containerID="fbc0e09e4c14fcc4bb456afa28885326699ee7f3b4d68f61696777fb5d221a77" exitCode=0
Sep 16 21:27:01.340120 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.340059    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl" event={"ID":"09d49cf5-96c1-4308-b7da-7ff47d520ccd","Type":"ContainerDied","Data":"fbc0e09e4c14fcc4bb456afa28885326699ee7f3b4d68f61696777fb5d221a77"}
Sep 16 21:27:01.604827 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:01.604549    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"]
Sep 16 21:27:02.345736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:02.345697    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf" event={"ID":"611804fc-3070-49eb-b152-b2459ec900cb","Type":"ContainerStarted","Data":"f3ae0ac472a60e6552f62ce3f442927fd239b17476247c560d77c1e064184955"}
Sep 16 21:27:02.347948 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:02.347916    2567 generic.go:356] "Generic (PLEG): container finished" podID="09d49cf5-96c1-4308-b7da-7ff47d520ccd" containerID="9fcd96595ea726d5b4c440c4d8156dc1856f4c1bd7b9babfc8de5f061aeaab55" exitCode=0
Sep 16 21:27:02.348087 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:02.347952    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl" event={"ID":"09d49cf5-96c1-4308-b7da-7ff47d520ccd","Type":"ContainerDied","Data":"9fcd96595ea726d5b4c440c4d8156dc1856f4c1bd7b9babfc8de5f061aeaab55"}
Sep 16 21:27:04.853278 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:04.853254    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:27:04.887595 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:04.887568    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-bundle\") pod \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") "
Sep 16 21:27:04.887713 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:04.887618    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4scsn\" (UniqueName: \"kubernetes.io/projected/09d49cf5-96c1-4308-b7da-7ff47d520ccd-kube-api-access-4scsn\") pod \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") "
Sep 16 21:27:04.887713 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:04.887670    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-util\") pod \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\" (UID: \"09d49cf5-96c1-4308-b7da-7ff47d520ccd\") "
Sep 16 21:27:04.888634 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:04.888609    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-bundle" (OuterVolumeSpecName: "bundle") pod "09d49cf5-96c1-4308-b7da-7ff47d520ccd" (UID: "09d49cf5-96c1-4308-b7da-7ff47d520ccd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:27:04.890430 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:04.890385    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d49cf5-96c1-4308-b7da-7ff47d520ccd-kube-api-access-4scsn" (OuterVolumeSpecName: "kube-api-access-4scsn") pod "09d49cf5-96c1-4308-b7da-7ff47d520ccd" (UID: "09d49cf5-96c1-4308-b7da-7ff47d520ccd"). InnerVolumeSpecName "kube-api-access-4scsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:27:04.895542 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:04.895517    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-util" (OuterVolumeSpecName: "util") pod "09d49cf5-96c1-4308-b7da-7ff47d520ccd" (UID: "09d49cf5-96c1-4308-b7da-7ff47d520ccd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:27:04.988936 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:04.988914    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-bundle\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:27:04.988936 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:04.988938    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4scsn\" (UniqueName: \"kubernetes.io/projected/09d49cf5-96c1-4308-b7da-7ff47d520ccd-kube-api-access-4scsn\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:27:04.989056 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:04.988947    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d49cf5-96c1-4308-b7da-7ff47d520ccd-util\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:27:05.364498 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:05.364466    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl"
Sep 16 21:27:05.364705 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:05.364468    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cfszl" event={"ID":"09d49cf5-96c1-4308-b7da-7ff47d520ccd","Type":"ContainerDied","Data":"df069ac2c01f58245e2ffba6a1b3db824dc57c003e025df943e68075ee7eec7c"}
Sep 16 21:27:05.364705 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:05.364634    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df069ac2c01f58245e2ffba6a1b3db824dc57c003e025df943e68075ee7eec7c"
Sep 16 21:27:05.366309 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:05.366282    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf" event={"ID":"611804fc-3070-49eb-b152-b2459ec900cb","Type":"ContainerStarted","Data":"c5e8647cffd6f8ad388925c5e642ff52775d4fae17673afa7fe8c4da627a7ac4"}
Sep 16 21:27:06.372365 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:06.372281    2567 generic.go:356] "Generic (PLEG): container finished" podID="611804fc-3070-49eb-b152-b2459ec900cb" containerID="c5e8647cffd6f8ad388925c5e642ff52775d4fae17673afa7fe8c4da627a7ac4" exitCode=0
Sep 16 21:27:06.372726 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:06.372371    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf" event={"ID":"611804fc-3070-49eb-b152-b2459ec900cb","Type":"ContainerDied","Data":"c5e8647cffd6f8ad388925c5e642ff52775d4fae17673afa7fe8c4da627a7ac4"}
Sep 16 21:27:07.379557 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:07.379520    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf" event={"ID":"611804fc-3070-49eb-b152-b2459ec900cb","Type":"ContainerStarted","Data":"dc0f7de930bf08931e1e65e3dafb80d88c8232c9957177d2f16d400520e79388"}
Sep 16 21:27:07.380028 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:07.379619    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:07.402057 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:07.401992    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf" podStartSLOduration=2.622826558 podStartE2EDuration="7.401977473s" totalImagesPullingTime="3.687760299s" totalInitContainerRuntime="1.091390616s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:07.400054586 +0000 UTC m=+266.785737724" watchObservedRunningTime="2026-09-16 21:27:07.401977473 +0000 UTC m=+266.787660556"
Sep 16 21:27:18.385466 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:18.385436    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dpdnf"
Sep 16 21:27:20.359541 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.359507    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"]
Sep 16 21:27:20.359899 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.359799    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d49cf5-96c1-4308-b7da-7ff47d520ccd" containerName="util"
Sep 16 21:27:20.359899 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.359810    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d49cf5-96c1-4308-b7da-7ff47d520ccd" containerName="util"
Sep 16 21:27:20.359899 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.359823    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d49cf5-96c1-4308-b7da-7ff47d520ccd" containerName="pull"
Sep 16 21:27:20.359899 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.359828    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d49cf5-96c1-4308-b7da-7ff47d520ccd" containerName="pull"
Sep 16 21:27:20.359899 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.359843    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d49cf5-96c1-4308-b7da-7ff47d520ccd" containerName="extract"
Sep 16 21:27:20.359899 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.359848    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d49cf5-96c1-4308-b7da-7ff47d520ccd" containerName="extract"
Sep 16 21:27:20.359899 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.359892    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d49cf5-96c1-4308-b7da-7ff47d520ccd" containerName="extract"
Sep 16 21:27:20.364951 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.364930    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:20.367335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.367312    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 21:27:20.368001 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.367985    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8f4r8\""
Sep 16 21:27:20.368089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.368041    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 21:27:20.368089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.368040    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 21:27:20.368204 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.368060    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 21:27:20.379615 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.379590    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"]
Sep 16 21:27:20.414416 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.414377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-tls-certs\") pod \"kube-auth-proxy-7f65bc968c-vt5vn\" (UID: \"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:20.414521 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.414429    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-tmp\") pod \"kube-auth-proxy-7f65bc968c-vt5vn\" (UID: \"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:20.414521 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.414469    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9kt\" (UniqueName: \"kubernetes.io/projected/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-kube-api-access-4l9kt\") pod \"kube-auth-proxy-7f65bc968c-vt5vn\" (UID: \"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:20.515243 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.515212    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-tls-certs\") pod \"kube-auth-proxy-7f65bc968c-vt5vn\" (UID: \"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:20.515243 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.515245    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-tmp\") pod \"kube-auth-proxy-7f65bc968c-vt5vn\" (UID: \"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:20.515455 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:27:20.515339    2567 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 16 21:27:20.515455 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.515370    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9kt\" (UniqueName: \"kubernetes.io/projected/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-kube-api-access-4l9kt\") pod \"kube-auth-proxy-7f65bc968c-vt5vn\" (UID: \"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:20.515455 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:27:20.515391    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-tls-certs podName:ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd nodeName:}" failed. No retries permitted until 2026-09-16 21:27:21.015375588 +0000 UTC m=+280.401058650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-tls-certs") pod "kube-auth-proxy-7f65bc968c-vt5vn" (UID: "ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd") : secret "kube-auth-proxy-tls" not found
Sep 16 21:27:20.517606 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.517581    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-tmp\") pod \"kube-auth-proxy-7f65bc968c-vt5vn\" (UID: \"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:20.523413 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:20.523379    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9kt\" (UniqueName: \"kubernetes.io/projected/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-kube-api-access-4l9kt\") pod \"kube-auth-proxy-7f65bc968c-vt5vn\" (UID: \"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:21.019213 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.019182    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-tls-certs\") pod \"kube-auth-proxy-7f65bc968c-vt5vn\" (UID: \"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:21.021715 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.021682    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd-tls-certs\") pod \"kube-auth-proxy-7f65bc968c-vt5vn\" (UID: \"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:21.280175 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.280095    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"
Sep 16 21:27:21.557627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.557602    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn"]
Sep 16 21:27:21.559566 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:27:21.559539    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4ca6d9_cadd_4f8a_ac97_300d78b63ecd.slice/crio-255944237d36b5563d2c7134780b463b7c69b57674f21eabd26e674715dd5e7e WatchSource:0}: Error finding container 255944237d36b5563d2c7134780b463b7c69b57674f21eabd26e674715dd5e7e: Status 404 returned error can't find the container with id 255944237d36b5563d2c7134780b463b7c69b57674f21eabd26e674715dd5e7e
Sep 16 21:27:21.632037 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.632003    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"]
Sep 16 21:27:21.634920 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.634900    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.637717 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.637699    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 21:27:21.637994 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.637977    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 21:27:21.638227 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.638213    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 21:27:21.638560 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.638547    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 21:27:21.638722 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.638711    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-np7bw\""
Sep 16 21:27:21.648172 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.648149    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"]
Sep 16 21:27:21.725909 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.725887    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.726022 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.725916    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj7cq\" (UniqueName: \"kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-kube-api-access-tj7cq\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.726022 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.725934    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.726022 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.725949    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.726151 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.726054    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.726151 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.726091    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.726151 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.726117    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.826516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.826447    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.826516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.826488    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj7cq\" (UniqueName: \"kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-kube-api-access-tj7cq\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.826516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.826509    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.826695 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.826529    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.826695 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.826631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.826695 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.826673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.826851 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.826706    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.827357 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.827333    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.828900 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.828874    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.829001 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.828906    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.829001 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.828917    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.829113 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.829028    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.834814 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.834791    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.835037 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.835018    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj7cq\" (UniqueName: \"kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-kube-api-access-tj7cq\") pod \"istiod-openshift-gateway-7b7995d5cd-r7l2b\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:21.944818 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:21.944796    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:22.221603 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:22.221575    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"]
Sep 16 21:27:22.223824 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:27:22.223798    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9161b0_08fb_4f7b_9b9f_4acd29dbd838.slice/crio-9c5b96dfe02517268efcbfffb6a133701ef5601f1602ac65016890b810855854 WatchSource:0}: Error finding container 9c5b96dfe02517268efcbfffb6a133701ef5601f1602ac65016890b810855854: Status 404 returned error can't find the container with id 9c5b96dfe02517268efcbfffb6a133701ef5601f1602ac65016890b810855854
Sep 16 21:27:22.448826 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:22.448783    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b" event={"ID":"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838","Type":"ContainerStarted","Data":"9c5b96dfe02517268efcbfffb6a133701ef5601f1602ac65016890b810855854"}
Sep 16 21:27:22.450562 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:22.450527    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn" event={"ID":"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd","Type":"ContainerStarted","Data":"255944237d36b5563d2c7134780b463b7c69b57674f21eabd26e674715dd5e7e"}
Sep 16 21:27:25.667855 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:25.667807    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:27:25.668120 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:25.667894    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:27:26.478316 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:26.478275    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b" event={"ID":"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838","Type":"ContainerStarted","Data":"0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859"}
Sep 16 21:27:26.478652 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:26.478632    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:26.480988 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:26.480941    2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-r7l2b container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 21:27:26.481126 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:26.481013    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b" podUID="4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 21:27:26.483078 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:26.483053    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn" event={"ID":"ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd","Type":"ContainerStarted","Data":"ef31ca40d88c77447a2e5c6c61b5b8921f452c42532bdc2378fd4c1e1af4250a"}
Sep 16 21:27:26.499094 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:26.499046    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b" podStartSLOduration=2.057494432 podStartE2EDuration="5.499032437s" totalImagesPullingTime="3.441538005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:26.496602749 +0000 UTC m=+285.882285854" watchObservedRunningTime="2026-09-16 21:27:26.499032437 +0000 UTC m=+285.884715521"
Sep 16 21:27:26.511795 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:26.511755    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f65bc968c-vt5vn" podStartSLOduration=2.40371996 podStartE2EDuration="6.511738547s" totalImagesPullingTime="4.108018587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:26.510300897 +0000 UTC m=+285.895984000" watchObservedRunningTime="2026-09-16 21:27:26.511738547 +0000 UTC m=+285.897421632"
Sep 16 21:27:27.488085 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:27.488058    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:38.806185 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:38.806150    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"]
Sep 16 21:27:38.813880 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:38.813857    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:38.822414 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:38.822374    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"]
Sep 16 21:27:38.967943 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:38.967914    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67d01f03-a6c2-4075-8999-9a667273878d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:38.968073 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:38.967951    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67d01f03-a6c2-4075-8999-9a667273878d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:38.968073 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:38.967973    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67d01f03-a6c2-4075-8999-9a667273878d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:38.968073 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:38.968049    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d01f03-a6c2-4075-8999-9a667273878d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:38.968190 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:38.968076    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67d01f03-a6c2-4075-8999-9a667273878d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:38.968190 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:38.968097    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d01f03-a6c2-4075-8999-9a667273878d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:38.968190 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:38.968160    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfwmx\" (UniqueName: \"kubernetes.io/projected/67d01f03-a6c2-4075-8999-9a667273878d-kube-api-access-kfwmx\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.069204 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.069138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67d01f03-a6c2-4075-8999-9a667273878d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.069204 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.069178    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d01f03-a6c2-4075-8999-9a667273878d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.069204 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.069196    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67d01f03-a6c2-4075-8999-9a667273878d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.069482 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.069223    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d01f03-a6c2-4075-8999-9a667273878d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.069482 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.069272    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfwmx\" (UniqueName: \"kubernetes.io/projected/67d01f03-a6c2-4075-8999-9a667273878d-kube-api-access-kfwmx\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.069482 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.069341    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67d01f03-a6c2-4075-8999-9a667273878d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.069482 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.069369    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67d01f03-a6c2-4075-8999-9a667273878d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.070090 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.070067    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67d01f03-a6c2-4075-8999-9a667273878d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.071578 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.071553    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d01f03-a6c2-4075-8999-9a667273878d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.071676 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.071577    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67d01f03-a6c2-4075-8999-9a667273878d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.071735 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.071674    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67d01f03-a6c2-4075-8999-9a667273878d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.071943 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.071923    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67d01f03-a6c2-4075-8999-9a667273878d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.083752 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.083730    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfwmx\" (UniqueName: \"kubernetes.io/projected/67d01f03-a6c2-4075-8999-9a667273878d-kube-api-access-kfwmx\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.087695 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.087667    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d01f03-a6c2-4075-8999-9a667273878d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xcdvq\" (UID: \"67d01f03-a6c2-4075-8999-9a667273878d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.124300 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.124280    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.400438 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.400412    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"]
Sep 16 21:27:39.402872 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:27:39.402845    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d01f03_a6c2_4075_8999_9a667273878d.slice/crio-3644c8fca1192569ee25fd4e8db082fd5313bc3268a46f7dcc11a376d29552be WatchSource:0}: Error finding container 3644c8fca1192569ee25fd4e8db082fd5313bc3268a46f7dcc11a376d29552be: Status 404 returned error can't find the container with id 3644c8fca1192569ee25fd4e8db082fd5313bc3268a46f7dcc11a376d29552be
Sep 16 21:27:39.406019 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.405986    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:27:39.406144 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.406061    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:27:39.544981 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.544941    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq" event={"ID":"67d01f03-a6c2-4075-8999-9a667273878d","Type":"ContainerStarted","Data":"cedc701ecfc77193cd7158a3c094f562743dbbdf9e00bf9449861eda28d0a0b9"}
Sep 16 21:27:39.545167 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.545018    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:39.545167 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.545040    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq" event={"ID":"67d01f03-a6c2-4075-8999-9a667273878d","Type":"ContainerStarted","Data":"3644c8fca1192569ee25fd4e8db082fd5313bc3268a46f7dcc11a376d29552be"}
Sep 16 21:27:39.585616 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:39.585376    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq" podStartSLOduration=1.58535468 podStartE2EDuration="1.58535468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:39.584491669 +0000 UTC m=+298.970174775" watchObservedRunningTime="2026-09-16 21:27:39.58535468 +0000 UTC m=+298.971037765"
Sep 16 21:27:40.551960 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.551924    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xcdvq"
Sep 16 21:27:40.632484 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.632451    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"]
Sep 16 21:27:40.632650 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.632631    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b" podUID="4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" containerName="discovery" containerID="cri-o://0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859" gracePeriod=30
Sep 16 21:27:40.877002 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.876979    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:40.985486 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.985456    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-dns-cert\") pod \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") "
Sep 16 21:27:40.985660 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.985509    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj7cq\" (UniqueName: \"kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-kube-api-access-tj7cq\") pod \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") "
Sep 16 21:27:40.985660 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.985568    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-kubeconfig\") pod \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") "
Sep 16 21:27:40.985660 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.985590    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-token\") pod \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") "
Sep 16 21:27:40.985660 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.985627    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-cacerts\") pod \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") "
Sep 16 21:27:40.985946 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.985726    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-ca-configmap\") pod \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") "
Sep 16 21:27:40.985946 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.985762    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-local-certs\") pod \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\" (UID: \"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838\") "
Sep 16 21:27:40.988663 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.988611    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" (UID: "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:27:40.988780 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.988670    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" (UID: "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:27:40.994579 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.991557    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-cacerts" (OuterVolumeSpecName: "cacerts") pod "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" (UID: "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:27:40.994579 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.991595    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" (UID: "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:27:40.994579 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.991637    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-local-certs" (OuterVolumeSpecName: "local-certs") pod "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" (UID: "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:27:40.994579 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.991661    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" (UID: "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:27:40.994579 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:40.992357    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-kube-api-access-tj7cq" (OuterVolumeSpecName: "kube-api-access-tj7cq") pod "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" (UID: "4b9161b0-08fb-4f7b-9b9f-4acd29dbd838"). InnerVolumeSpecName "kube-api-access-tj7cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:27:41.014961 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.014938    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 21:27:41.087406 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.087248    2567 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-kubeconfig\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:27:41.087406 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.087270    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-token\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:27:41.087406 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.087279    2567 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-cacerts\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:27:41.087406 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.087287    2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-ca-configmap\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:27:41.087406 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.087296    2567 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-local-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:27:41.087406 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.087306    2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-istio-csr-dns-cert\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:27:41.087406 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.087314    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj7cq\" (UniqueName: \"kubernetes.io/projected/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838-kube-api-access-tj7cq\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:27:41.556715 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.556684    2567 generic.go:356] "Generic (PLEG): container finished" podID="4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" containerID="0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859" exitCode=0
Sep 16 21:27:41.557089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.556739    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"
Sep 16 21:27:41.557089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.556759    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b" event={"ID":"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838","Type":"ContainerDied","Data":"0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859"}
Sep 16 21:27:41.557089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.556818    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b" event={"ID":"4b9161b0-08fb-4f7b-9b9f-4acd29dbd838","Type":"ContainerDied","Data":"9c5b96dfe02517268efcbfffb6a133701ef5601f1602ac65016890b810855854"}
Sep 16 21:27:41.557089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.556851    2567 scope.go:117] "RemoveContainer" containerID="0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859"
Sep 16 21:27:41.566241 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.566166    2567 scope.go:117] "RemoveContainer" containerID="0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859"
Sep 16 21:27:41.566548 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:27:41.566496    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859\": container with ID starting with 0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859 not found: ID does not exist" containerID="0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859"
Sep 16 21:27:41.566548 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.566536    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859"} err="failed to get container status \"0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859\": rpc error: code = NotFound desc = could not find container \"0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859\": container with ID starting with 0a82f0b77d49d0e0b6ae2d959e8739268652f3aa17408b92921a5c3bb3f5e859 not found: ID does not exist"
Sep 16 21:27:41.606584 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.606558    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"]
Sep 16 21:27:41.614667 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.614642    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-r7l2b"]
Sep 16 21:27:41.808459 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.808359    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"]
Sep 16 21:27:41.808710 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.808696    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" containerName="discovery"
Sep 16 21:27:41.808756 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.808712    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" containerName="discovery"
Sep 16 21:27:41.808793 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.808775    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" containerName="discovery"
Sep 16 21:27:41.811820 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.811804    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"
Sep 16 21:27:41.815532 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.815509    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zzt7n\""
Sep 16 21:27:41.815749 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.815725    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 21:27:41.829932 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.829904    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"]
Sep 16 21:27:41.832101 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.832078    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"]
Sep 16 21:27:41.832211 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.832170    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"
Sep 16 21:27:41.834707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.834688    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 21:27:41.834794 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.834690    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-77pmk\""
Sep 16 21:27:41.853869 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.853844    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"]
Sep 16 21:27:41.894578 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.894548    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9c99\" (UniqueName: \"kubernetes.io/projected/272ac6f7-a92e-45c8-869d-c6a9b4bc9682-kube-api-access-r9c99\") pod \"kserve-controller-manager-d6f4b65b8-wvrjt\" (UID: \"272ac6f7-a92e-45c8-869d-c6a9b4bc9682\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"
Sep 16 21:27:41.894703 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.894606    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/272ac6f7-a92e-45c8-869d-c6a9b4bc9682-cert\") pod \"kserve-controller-manager-d6f4b65b8-wvrjt\" (UID: \"272ac6f7-a92e-45c8-869d-c6a9b4bc9682\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"
Sep 16 21:27:41.995704 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.995677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/272ac6f7-a92e-45c8-869d-c6a9b4bc9682-cert\") pod \"kserve-controller-manager-d6f4b65b8-wvrjt\" (UID: \"272ac6f7-a92e-45c8-869d-c6a9b4bc9682\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"
Sep 16 21:27:41.995839 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.995724    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8hz\" (UniqueName: \"kubernetes.io/projected/da9e2bd5-c84e-494f-815e-135e7d3dc26b-kube-api-access-lk8hz\") pod \"llmisvc-controller-manager-777c6bd7d5-tbpfn\" (UID: \"da9e2bd5-c84e-494f-815e-135e7d3dc26b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"
Sep 16 21:27:41.995839 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.995761    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9c99\" (UniqueName: \"kubernetes.io/projected/272ac6f7-a92e-45c8-869d-c6a9b4bc9682-kube-api-access-r9c99\") pod \"kserve-controller-manager-d6f4b65b8-wvrjt\" (UID: \"272ac6f7-a92e-45c8-869d-c6a9b4bc9682\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"
Sep 16 21:27:41.995839 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:41.995782    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da9e2bd5-c84e-494f-815e-135e7d3dc26b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tbpfn\" (UID: \"da9e2bd5-c84e-494f-815e-135e7d3dc26b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"
Sep 16 21:27:41.995839 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:27:41.995827    2567 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 21:27:41.996031 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:27:41.995885    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272ac6f7-a92e-45c8-869d-c6a9b4bc9682-cert podName:272ac6f7-a92e-45c8-869d-c6a9b4bc9682 nodeName:}" failed. No retries permitted until 2026-09-16 21:27:42.495869246 +0000 UTC m=+301.881552311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/272ac6f7-a92e-45c8-869d-c6a9b4bc9682-cert") pod "kserve-controller-manager-d6f4b65b8-wvrjt" (UID: "272ac6f7-a92e-45c8-869d-c6a9b4bc9682") : secret "kserve-webhook-server-cert" not found
Sep 16 21:27:42.014449 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:42.014424    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9c99\" (UniqueName: \"kubernetes.io/projected/272ac6f7-a92e-45c8-869d-c6a9b4bc9682-kube-api-access-r9c99\") pod \"kserve-controller-manager-d6f4b65b8-wvrjt\" (UID: \"272ac6f7-a92e-45c8-869d-c6a9b4bc9682\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"
Sep 16 21:27:42.096424 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:42.096330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8hz\" (UniqueName: \"kubernetes.io/projected/da9e2bd5-c84e-494f-815e-135e7d3dc26b-kube-api-access-lk8hz\") pod \"llmisvc-controller-manager-777c6bd7d5-tbpfn\" (UID: \"da9e2bd5-c84e-494f-815e-135e7d3dc26b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"
Sep 16 21:27:42.096424 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:42.096386    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da9e2bd5-c84e-494f-815e-135e7d3dc26b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tbpfn\" (UID: \"da9e2bd5-c84e-494f-815e-135e7d3dc26b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"
Sep 16 21:27:42.096595 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:27:42.096491    2567 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 21:27:42.096595 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:27:42.096538    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9e2bd5-c84e-494f-815e-135e7d3dc26b-cert podName:da9e2bd5-c84e-494f-815e-135e7d3dc26b nodeName:}" failed. No retries permitted until 2026-09-16 21:27:42.596525803 +0000 UTC m=+301.982208865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da9e2bd5-c84e-494f-815e-135e7d3dc26b-cert") pod "llmisvc-controller-manager-777c6bd7d5-tbpfn" (UID: "da9e2bd5-c84e-494f-815e-135e7d3dc26b") : secret "llmisvc-webhook-server-cert" not found
Sep 16 21:27:42.119741 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:42.119714    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8hz\" (UniqueName: \"kubernetes.io/projected/da9e2bd5-c84e-494f-815e-135e7d3dc26b-kube-api-access-lk8hz\") pod \"llmisvc-controller-manager-777c6bd7d5-tbpfn\" (UID: \"da9e2bd5-c84e-494f-815e-135e7d3dc26b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"
Sep 16 21:27:42.499650 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:42.499605    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/272ac6f7-a92e-45c8-869d-c6a9b4bc9682-cert\") pod \"kserve-controller-manager-d6f4b65b8-wvrjt\" (UID: \"272ac6f7-a92e-45c8-869d-c6a9b4bc9682\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"
Sep 16 21:27:42.502090 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:42.502068    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/272ac6f7-a92e-45c8-869d-c6a9b4bc9682-cert\") pod \"kserve-controller-manager-d6f4b65b8-wvrjt\" (UID: \"272ac6f7-a92e-45c8-869d-c6a9b4bc9682\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"
Sep 16 21:27:42.600733 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:42.600702    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da9e2bd5-c84e-494f-815e-135e7d3dc26b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tbpfn\" (UID: \"da9e2bd5-c84e-494f-815e-135e7d3dc26b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"
Sep 16 21:27:42.603021 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:42.603002    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da9e2bd5-c84e-494f-815e-135e7d3dc26b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tbpfn\" (UID: \"da9e2bd5-c84e-494f-815e-135e7d3dc26b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"
Sep 16 21:27:42.721055 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:42.721024    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"
Sep 16 21:27:42.742135 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:42.742108    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"
Sep 16 21:27:43.011213 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:27:43.011185    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272ac6f7_a92e_45c8_869d_c6a9b4bc9682.slice/crio-e3eb48869395c1e883a5e8c7194ea4246b1eaf1d886a02fdc76a8b9077fcc5dd WatchSource:0}: Error finding container e3eb48869395c1e883a5e8c7194ea4246b1eaf1d886a02fdc76a8b9077fcc5dd: Status 404 returned error can't find the container with id e3eb48869395c1e883a5e8c7194ea4246b1eaf1d886a02fdc76a8b9077fcc5dd
Sep 16 21:27:43.012641 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:43.012622    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:27:43.023599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:43.023576    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"]
Sep 16 21:27:43.043180 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:43.043157    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"]
Sep 16 21:27:43.044586 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:27:43.044566    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podda9e2bd5_c84e_494f_815e_135e7d3dc26b.slice/crio-13702ae05d99253fc184dbfeea5479e8a3a888ba3cbfce31acdc1d608b9056db WatchSource:0}: Error finding container 13702ae05d99253fc184dbfeea5479e8a3a888ba3cbfce31acdc1d608b9056db: Status 404 returned error can't find the container with id 13702ae05d99253fc184dbfeea5479e8a3a888ba3cbfce31acdc1d608b9056db
Sep 16 21:27:43.109269 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:43.109228    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b9161b0-08fb-4f7b-9b9f-4acd29dbd838" path="/var/lib/kubelet/pods/4b9161b0-08fb-4f7b-9b9f-4acd29dbd838/volumes"
Sep 16 21:27:43.574485 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:43.574435    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn" event={"ID":"da9e2bd5-c84e-494f-815e-135e7d3dc26b","Type":"ContainerStarted","Data":"13702ae05d99253fc184dbfeea5479e8a3a888ba3cbfce31acdc1d608b9056db"}
Sep 16 21:27:43.576124 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:43.576092    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt" event={"ID":"272ac6f7-a92e-45c8-869d-c6a9b4bc9682","Type":"ContainerStarted","Data":"e3eb48869395c1e883a5e8c7194ea4246b1eaf1d886a02fdc76a8b9077fcc5dd"}
Sep 16 21:27:46.594452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:46.594405    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn" event={"ID":"da9e2bd5-c84e-494f-815e-135e7d3dc26b","Type":"ContainerStarted","Data":"c690bf4ec5deca3ac9cd0506157bbbfc16e90d740cd942703e6e15e1bc2d68c1"}
Sep 16 21:27:46.594894 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:46.594572    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"
Sep 16 21:27:46.595949 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:46.595925    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt" event={"ID":"272ac6f7-a92e-45c8-869d-c6a9b4bc9682","Type":"ContainerStarted","Data":"7ea00b6878681a96fa35289a5d36948bd60c75e3391ee117dd6a19fca56c2668"}
Sep 16 21:27:46.596059 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:46.595972    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"
Sep 16 21:27:46.609854 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:46.609816    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn" podStartSLOduration=2.607538295 podStartE2EDuration="5.609802714s" totalImagesPullingTime="3.002264419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:46.608433285 +0000 UTC m=+305.994116380" watchObservedRunningTime="2026-09-16 21:27:46.609802714 +0000 UTC m=+305.995485846"
Sep 16 21:27:46.623796 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:27:46.623757    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt" podStartSLOduration=3.138310841 podStartE2EDuration="5.623744673s" totalImagesPullingTime="2.485433832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:46.622156492 +0000 UTC m=+306.007839576" watchObservedRunningTime="2026-09-16 21:27:46.623744673 +0000 UTC m=+306.009427776"
Sep 16 21:28:17.604273 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:17.604239    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbpfn"
Sep 16 21:28:17.607546 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:17.607524    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wvrjt"
Sep 16 21:28:47.214167 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.214132    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"]
Sep 16 21:28:47.217278 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.217257    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:47.219530 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.219506    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 21:28:47.220229 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.220203    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 21:28:47.220229 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.220217    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-shrwc\""
Sep 16 21:28:47.227457 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.227437    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"]
Sep 16 21:28:47.273303 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.273273    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"]
Sep 16 21:28:47.275736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.275717    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:47.286074 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.286052    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"]
Sep 16 21:28:47.297381 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.297361    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:47.297516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.297479    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:47.297583 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.297529    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b44gh\" (UniqueName: \"kubernetes.io/projected/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-kube-api-access-b44gh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:47.359469 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.359442    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"]
Sep 16 21:28:47.362016 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.361999    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:47.371025 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.371003    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"]
Sep 16 21:28:47.398319 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.398298    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zt74\" (UniqueName: \"kubernetes.io/projected/9c50f102-4948-4806-b084-6c63693493fd-kube-api-access-9zt74\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:47.398440 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.398330    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:47.398440 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.398364    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:47.398440 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.398384    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:47.398548 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.398486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:47.398548 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.398523    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b44gh\" (UniqueName: \"kubernetes.io/projected/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-kube-api-access-b44gh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:47.398736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.398716    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:47.398778 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.398767    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:47.405718 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.405697    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b44gh\" (UniqueName: \"kubernetes.io/projected/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-kube-api-access-b44gh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:47.460789 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.460766    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"]
Sep 16 21:28:47.463327 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.463311    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:47.471766 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.471746    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"]
Sep 16 21:28:47.499059 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.499034    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:47.499180 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.499062    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cttd4\" (UniqueName: \"kubernetes.io/projected/e7423c65-e590-496c-a39c-a053f36c0f80-kube-api-access-cttd4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:47.499180 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.499080    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:47.499180 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.499157    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zt74\" (UniqueName: \"kubernetes.io/projected/9c50f102-4948-4806-b084-6c63693493fd-kube-api-access-9zt74\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:47.499285 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.499185    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:47.499285 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.499217    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:47.499607 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.499588    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:47.499648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.499602    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:47.506546 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.506527    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zt74\" (UniqueName: \"kubernetes.io/projected/9c50f102-4948-4806-b084-6c63693493fd-kube-api-access-9zt74\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:47.526329 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.526307    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:47.584871 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.584835    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:47.600324 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.600224    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:47.600462 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.600327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnkt\" (UniqueName: \"kubernetes.io/projected/df4e8739-6f71-4af4-a728-4f3e6a85a9df-kube-api-access-9nnkt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:47.600462 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.600364    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:47.600462 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.600446    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:47.600631 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.600481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttd4\" (UniqueName: \"kubernetes.io/projected/e7423c65-e590-496c-a39c-a053f36c0f80-kube-api-access-cttd4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:47.600631 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.600508    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:47.600916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.600888    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:47.601035 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.601014    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:47.608755 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.608723    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cttd4\" (UniqueName: \"kubernetes.io/projected/e7423c65-e590-496c-a39c-a053f36c0f80-kube-api-access-cttd4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:47.671858 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.671831    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:47.701649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.701621    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnkt\" (UniqueName: \"kubernetes.io/projected/df4e8739-6f71-4af4-a728-4f3e6a85a9df-kube-api-access-9nnkt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:47.701774 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.701661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:47.701774 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.701707    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:47.702052 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.702029    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:47.702186 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.702165    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:47.710347 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.710326    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnkt\" (UniqueName: \"kubernetes.io/projected/df4e8739-6f71-4af4-a728-4f3e6a85a9df-kube-api-access-9nnkt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:47.772901 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.772873    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:47.796135 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.796113    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"]
Sep 16 21:28:47.797967 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:28:47.797945    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1440fda4_9c7e_4740_a5f6_e9bc20dbece8.slice/crio-488c6f214f08e6054c4ba42e5a26c69b65097a47fc857be0e6cbfc804001e125 WatchSource:0}: Error finding container 488c6f214f08e6054c4ba42e5a26c69b65097a47fc857be0e6cbfc804001e125: Status 404 returned error can't find the container with id 488c6f214f08e6054c4ba42e5a26c69b65097a47fc857be0e6cbfc804001e125
Sep 16 21:28:47.859914 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.859892    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"]
Sep 16 21:28:47.861874 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:28:47.861839    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c50f102_4948_4806_b084_6c63693493fd.slice/crio-4bcd8e43a6afeaa9920871d314c20392727085714b71891095ccf895e9586507 WatchSource:0}: Error finding container 4bcd8e43a6afeaa9920871d314c20392727085714b71891095ccf895e9586507: Status 404 returned error can't find the container with id 4bcd8e43a6afeaa9920871d314c20392727085714b71891095ccf895e9586507
Sep 16 21:28:47.888876 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.888847    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm" event={"ID":"9c50f102-4948-4806-b084-6c63693493fd","Type":"ContainerStarted","Data":"4bcd8e43a6afeaa9920871d314c20392727085714b71891095ccf895e9586507"}
Sep 16 21:28:47.890496 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.890472    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp" event={"ID":"1440fda4-9c7e-4740-a5f6-e9bc20dbece8","Type":"ContainerStarted","Data":"7939b7e4fc1b827ee63c56479e852eeef0b3f0ac779e75576bcfcc6e89e337ae"}
Sep 16 21:28:47.890591 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.890518    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp" event={"ID":"1440fda4-9c7e-4740-a5f6-e9bc20dbece8","Type":"ContainerStarted","Data":"488c6f214f08e6054c4ba42e5a26c69b65097a47fc857be0e6cbfc804001e125"}
Sep 16 21:28:47.941978 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:47.941956    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"]
Sep 16 21:28:47.956178 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:28:47.956152    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7423c65_e590_496c_a39c_a053f36c0f80.slice/crio-73566926d5af0e3f8c7f174283657c4d301729ffb8a61d4b93f782275425bccf WatchSource:0}: Error finding container 73566926d5af0e3f8c7f174283657c4d301729ffb8a61d4b93f782275425bccf: Status 404 returned error can't find the container with id 73566926d5af0e3f8c7f174283657c4d301729ffb8a61d4b93f782275425bccf
Sep 16 21:28:48.049003 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:48.048973    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"]
Sep 16 21:28:48.049567 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:28:48.049539    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4e8739_6f71_4af4_a728_4f3e6a85a9df.slice/crio-af84b1d033ea38be2d80133fc157c868963102d309631ead4341d68c89918050 WatchSource:0}: Error finding container af84b1d033ea38be2d80133fc157c868963102d309631ead4341d68c89918050: Status 404 returned error can't find the container with id af84b1d033ea38be2d80133fc157c868963102d309631ead4341d68c89918050
Sep 16 21:28:48.896588 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:48.896552    2567 generic.go:356] "Generic (PLEG): container finished" podID="9c50f102-4948-4806-b084-6c63693493fd" containerID="c8e6569c1db77866b1a25f0ad88d0755832b873226634b206de88f41a623321f" exitCode=0
Sep 16 21:28:48.897252 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:48.896641    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm" event={"ID":"9c50f102-4948-4806-b084-6c63693493fd","Type":"ContainerDied","Data":"c8e6569c1db77866b1a25f0ad88d0755832b873226634b206de88f41a623321f"}
Sep 16 21:28:48.898283 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:48.898259    2567 generic.go:356] "Generic (PLEG): container finished" podID="1440fda4-9c7e-4740-a5f6-e9bc20dbece8" containerID="7939b7e4fc1b827ee63c56479e852eeef0b3f0ac779e75576bcfcc6e89e337ae" exitCode=0
Sep 16 21:28:48.898378 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:48.898354    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp" event={"ID":"1440fda4-9c7e-4740-a5f6-e9bc20dbece8","Type":"ContainerDied","Data":"7939b7e4fc1b827ee63c56479e852eeef0b3f0ac779e75576bcfcc6e89e337ae"}
Sep 16 21:28:48.899995 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:48.899976    2567 generic.go:356] "Generic (PLEG): container finished" podID="df4e8739-6f71-4af4-a728-4f3e6a85a9df" containerID="b27518ae6a2d619975306511376c5fab23b729cff40bf09232f79f7fd8445438" exitCode=0
Sep 16 21:28:48.900077 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:48.900053    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp" event={"ID":"df4e8739-6f71-4af4-a728-4f3e6a85a9df","Type":"ContainerDied","Data":"b27518ae6a2d619975306511376c5fab23b729cff40bf09232f79f7fd8445438"}
Sep 16 21:28:48.900143 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:48.900096    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp" event={"ID":"df4e8739-6f71-4af4-a728-4f3e6a85a9df","Type":"ContainerStarted","Data":"af84b1d033ea38be2d80133fc157c868963102d309631ead4341d68c89918050"}
Sep 16 21:28:48.901708 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:48.901681    2567 generic.go:356] "Generic (PLEG): container finished" podID="e7423c65-e590-496c-a39c-a053f36c0f80" containerID="14da9c7e9631bf397738df95f8bdefe88be9d8c73c12e8e7c7e3e92492b94a3b" exitCode=0
Sep 16 21:28:48.901807 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:48.901729    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp" event={"ID":"e7423c65-e590-496c-a39c-a053f36c0f80","Type":"ContainerDied","Data":"14da9c7e9631bf397738df95f8bdefe88be9d8c73c12e8e7c7e3e92492b94a3b"}
Sep 16 21:28:48.901807 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:48.901768    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp" event={"ID":"e7423c65-e590-496c-a39c-a053f36c0f80","Type":"ContainerStarted","Data":"73566926d5af0e3f8c7f174283657c4d301729ffb8a61d4b93f782275425bccf"}
Sep 16 21:28:49.912345 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:49.912312    2567 generic.go:356] "Generic (PLEG): container finished" podID="df4e8739-6f71-4af4-a728-4f3e6a85a9df" containerID="d75b61ba80639dfc7a1a276587b1e35d335b0b9e4bc9aabb01816853eaf8ec7a" exitCode=0
Sep 16 21:28:49.912699 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:49.912421    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp" event={"ID":"df4e8739-6f71-4af4-a728-4f3e6a85a9df","Type":"ContainerDied","Data":"d75b61ba80639dfc7a1a276587b1e35d335b0b9e4bc9aabb01816853eaf8ec7a"}
Sep 16 21:28:49.914525 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:49.914502    2567 generic.go:356] "Generic (PLEG): container finished" podID="e7423c65-e590-496c-a39c-a053f36c0f80" containerID="013384dff64c4380fd7d6c703b427cc34128a3522b3502d57fd17ec7268a0f54" exitCode=0
Sep 16 21:28:49.914622 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:49.914552    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp" event={"ID":"e7423c65-e590-496c-a39c-a053f36c0f80","Type":"ContainerDied","Data":"013384dff64c4380fd7d6c703b427cc34128a3522b3502d57fd17ec7268a0f54"}
Sep 16 21:28:50.921521 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:50.921484    2567 generic.go:356] "Generic (PLEG): container finished" podID="9c50f102-4948-4806-b084-6c63693493fd" containerID="9a837d043cbb0dc1d923878a5f57cf78437c272138d89cb3fa1fdfa7083b4195" exitCode=0
Sep 16 21:28:50.921902 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:50.921576    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm" event={"ID":"9c50f102-4948-4806-b084-6c63693493fd","Type":"ContainerDied","Data":"9a837d043cbb0dc1d923878a5f57cf78437c272138d89cb3fa1fdfa7083b4195"}
Sep 16 21:28:50.923643 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:50.923618    2567 generic.go:356] "Generic (PLEG): container finished" podID="1440fda4-9c7e-4740-a5f6-e9bc20dbece8" containerID="f3b25a62b28d287aca93afd5654fe5b5e3f28f08b09b800448ae71c2277545b0" exitCode=0
Sep 16 21:28:50.923753 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:50.923733    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp" event={"ID":"1440fda4-9c7e-4740-a5f6-e9bc20dbece8","Type":"ContainerDied","Data":"f3b25a62b28d287aca93afd5654fe5b5e3f28f08b09b800448ae71c2277545b0"}
Sep 16 21:28:50.926168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:50.926143    2567 generic.go:356] "Generic (PLEG): container finished" podID="df4e8739-6f71-4af4-a728-4f3e6a85a9df" containerID="2c6e442672d26e2d0812c9763aee0fbbb38274f17076ee663bc798c1c6e094e3" exitCode=0
Sep 16 21:28:50.926243 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:50.926220    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp" event={"ID":"df4e8739-6f71-4af4-a728-4f3e6a85a9df","Type":"ContainerDied","Data":"2c6e442672d26e2d0812c9763aee0fbbb38274f17076ee663bc798c1c6e094e3"}
Sep 16 21:28:50.928598 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:50.928575    2567 generic.go:356] "Generic (PLEG): container finished" podID="e7423c65-e590-496c-a39c-a053f36c0f80" containerID="1a72d958bf1ba4c6c51312ab28e05279566a129eb77e531bc07a533046549109" exitCode=0
Sep 16 21:28:50.928698 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:50.928631    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp" event={"ID":"e7423c65-e590-496c-a39c-a053f36c0f80","Type":"ContainerDied","Data":"1a72d958bf1ba4c6c51312ab28e05279566a129eb77e531bc07a533046549109"}
Sep 16 21:28:51.935100 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:51.935068    2567 generic.go:356] "Generic (PLEG): container finished" podID="1440fda4-9c7e-4740-a5f6-e9bc20dbece8" containerID="75bd4c62d012e926a8f759cabba5a044f8dfb12f0856c768829602cddbc1e737" exitCode=0
Sep 16 21:28:51.935536 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:51.935160    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp" event={"ID":"1440fda4-9c7e-4740-a5f6-e9bc20dbece8","Type":"ContainerDied","Data":"75bd4c62d012e926a8f759cabba5a044f8dfb12f0856c768829602cddbc1e737"}
Sep 16 21:28:51.937252 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:51.937223    2567 generic.go:356] "Generic (PLEG): container finished" podID="9c50f102-4948-4806-b084-6c63693493fd" containerID="1fc842da286fb16db3421abcef3fca85a93e6294688bdf6e288d164f21ba6c41" exitCode=0
Sep 16 21:28:51.937330 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:51.937281    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm" event={"ID":"9c50f102-4948-4806-b084-6c63693493fd","Type":"ContainerDied","Data":"1fc842da286fb16db3421abcef3fca85a93e6294688bdf6e288d164f21ba6c41"}
Sep 16 21:28:52.072856 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.072826    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:52.094070 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.094049    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:52.141141 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.141116    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-bundle\") pod \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") "
Sep 16 21:28:52.141233 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.141165    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nnkt\" (UniqueName: \"kubernetes.io/projected/df4e8739-6f71-4af4-a728-4f3e6a85a9df-kube-api-access-9nnkt\") pod \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") "
Sep 16 21:28:52.141233 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.141211    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-util\") pod \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\" (UID: \"df4e8739-6f71-4af4-a728-4f3e6a85a9df\") "
Sep 16 21:28:52.141741 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.141703    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-bundle" (OuterVolumeSpecName: "bundle") pod "df4e8739-6f71-4af4-a728-4f3e6a85a9df" (UID: "df4e8739-6f71-4af4-a728-4f3e6a85a9df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:28:52.143195 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.143170    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4e8739-6f71-4af4-a728-4f3e6a85a9df-kube-api-access-9nnkt" (OuterVolumeSpecName: "kube-api-access-9nnkt") pod "df4e8739-6f71-4af4-a728-4f3e6a85a9df" (UID: "df4e8739-6f71-4af4-a728-4f3e6a85a9df"). InnerVolumeSpecName "kube-api-access-9nnkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:28:52.148836 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.148814    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-util" (OuterVolumeSpecName: "util") pod "df4e8739-6f71-4af4-a728-4f3e6a85a9df" (UID: "df4e8739-6f71-4af4-a728-4f3e6a85a9df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:28:52.241600 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.241580    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-bundle\") pod \"e7423c65-e590-496c-a39c-a053f36c0f80\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") "
Sep 16 21:28:52.241702 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.241658    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cttd4\" (UniqueName: \"kubernetes.io/projected/e7423c65-e590-496c-a39c-a053f36c0f80-kube-api-access-cttd4\") pod \"e7423c65-e590-496c-a39c-a053f36c0f80\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") "
Sep 16 21:28:52.241702 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.241676    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-util\") pod \"e7423c65-e590-496c-a39c-a053f36c0f80\" (UID: \"e7423c65-e590-496c-a39c-a053f36c0f80\") "
Sep 16 21:28:52.241812 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.241797    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-util\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:52.241812 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.241807    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df4e8739-6f71-4af4-a728-4f3e6a85a9df-bundle\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:52.241913 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.241816    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nnkt\" (UniqueName: \"kubernetes.io/projected/df4e8739-6f71-4af4-a728-4f3e6a85a9df-kube-api-access-9nnkt\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:52.242075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.242055    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-bundle" (OuterVolumeSpecName: "bundle") pod "e7423c65-e590-496c-a39c-a053f36c0f80" (UID: "e7423c65-e590-496c-a39c-a053f36c0f80"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:28:52.243672 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.243652    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7423c65-e590-496c-a39c-a053f36c0f80-kube-api-access-cttd4" (OuterVolumeSpecName: "kube-api-access-cttd4") pod "e7423c65-e590-496c-a39c-a053f36c0f80" (UID: "e7423c65-e590-496c-a39c-a053f36c0f80"). InnerVolumeSpecName "kube-api-access-cttd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:28:52.246858 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.246820    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-util" (OuterVolumeSpecName: "util") pod "e7423c65-e590-496c-a39c-a053f36c0f80" (UID: "e7423c65-e590-496c-a39c-a053f36c0f80"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:28:52.342559 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.342539    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-bundle\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:52.342559 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.342558    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cttd4\" (UniqueName: \"kubernetes.io/projected/e7423c65-e590-496c-a39c-a053f36c0f80-kube-api-access-cttd4\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:52.342677 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.342568    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7423c65-e590-496c-a39c-a053f36c0f80-util\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:52.944776 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.944743    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp" event={"ID":"df4e8739-6f71-4af4-a728-4f3e6a85a9df","Type":"ContainerDied","Data":"af84b1d033ea38be2d80133fc157c868963102d309631ead4341d68c89918050"}
Sep 16 21:28:52.945209 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.944812    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af84b1d033ea38be2d80133fc157c868963102d309631ead4341d68c89918050"
Sep 16 21:28:52.945209 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.944754    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crxbhp"
Sep 16 21:28:52.947212 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.947181    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp"
Sep 16 21:28:52.947212 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.947200    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fx7kp" event={"ID":"e7423c65-e590-496c-a39c-a053f36c0f80","Type":"ContainerDied","Data":"73566926d5af0e3f8c7f174283657c4d301729ffb8a61d4b93f782275425bccf"}
Sep 16 21:28:52.947479 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:52.947235    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73566926d5af0e3f8c7f174283657c4d301729ffb8a61d4b93f782275425bccf"
Sep 16 21:28:53.078899 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.078869    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:53.105815 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.105794    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:53.149002 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.148978    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b44gh\" (UniqueName: \"kubernetes.io/projected/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-kube-api-access-b44gh\") pod \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") "
Sep 16 21:28:53.149133 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.149105    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-bundle\") pod \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") "
Sep 16 21:28:53.149133 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.149122    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-util\") pod \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\" (UID: \"1440fda4-9c7e-4740-a5f6-e9bc20dbece8\") "
Sep 16 21:28:53.149755 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.149726    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-bundle" (OuterVolumeSpecName: "bundle") pod "1440fda4-9c7e-4740-a5f6-e9bc20dbece8" (UID: "1440fda4-9c7e-4740-a5f6-e9bc20dbece8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:28:53.150990 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.150973    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-kube-api-access-b44gh" (OuterVolumeSpecName: "kube-api-access-b44gh") pod "1440fda4-9c7e-4740-a5f6-e9bc20dbece8" (UID: "1440fda4-9c7e-4740-a5f6-e9bc20dbece8"). InnerVolumeSpecName "kube-api-access-b44gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:28:53.187599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.187570    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-util" (OuterVolumeSpecName: "util") pod "1440fda4-9c7e-4740-a5f6-e9bc20dbece8" (UID: "1440fda4-9c7e-4740-a5f6-e9bc20dbece8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:28:53.249873 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.249850    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-bundle\") pod \"9c50f102-4948-4806-b084-6c63693493fd\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") "
Sep 16 21:28:53.249957 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.249906    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zt74\" (UniqueName: \"kubernetes.io/projected/9c50f102-4948-4806-b084-6c63693493fd-kube-api-access-9zt74\") pod \"9c50f102-4948-4806-b084-6c63693493fd\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") "
Sep 16 21:28:53.249957 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.249925    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-util\") pod \"9c50f102-4948-4806-b084-6c63693493fd\" (UID: \"9c50f102-4948-4806-b084-6c63693493fd\") "
Sep 16 21:28:53.250342 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.250307    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-bundle" (OuterVolumeSpecName: "bundle") pod "9c50f102-4948-4806-b084-6c63693493fd" (UID: "9c50f102-4948-4806-b084-6c63693493fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:28:53.250452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.250431    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-bundle\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:53.250452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.250448    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-util\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:53.250577 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.250463    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b44gh\" (UniqueName: \"kubernetes.io/projected/1440fda4-9c7e-4740-a5f6-e9bc20dbece8-kube-api-access-b44gh\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:53.250577 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.250478    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-bundle\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:53.252405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.252369    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c50f102-4948-4806-b084-6c63693493fd-kube-api-access-9zt74" (OuterVolumeSpecName: "kube-api-access-9zt74") pod "9c50f102-4948-4806-b084-6c63693493fd" (UID: "9c50f102-4948-4806-b084-6c63693493fd"). InnerVolumeSpecName "kube-api-access-9zt74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:28:53.255352 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.255312    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-util" (OuterVolumeSpecName: "util") pod "9c50f102-4948-4806-b084-6c63693493fd" (UID: "9c50f102-4948-4806-b084-6c63693493fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:28:53.351274 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.351251    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zt74\" (UniqueName: \"kubernetes.io/projected/9c50f102-4948-4806-b084-6c63693493fd-kube-api-access-9zt74\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:53.351274 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.351272    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c50f102-4948-4806-b084-6c63693493fd-util\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:28:53.954076 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.954039    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm"
Sep 16 21:28:53.954560 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.954075    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7qsm" event={"ID":"9c50f102-4948-4806-b084-6c63693493fd","Type":"ContainerDied","Data":"4bcd8e43a6afeaa9920871d314c20392727085714b71891095ccf895e9586507"}
Sep 16 21:28:53.954560 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.954123    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bcd8e43a6afeaa9920871d314c20392727085714b71891095ccf895e9586507"
Sep 16 21:28:53.956011 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.955992    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp"
Sep 16 21:28:53.956011 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.955998    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akwpdp" event={"ID":"1440fda4-9c7e-4740-a5f6-e9bc20dbece8","Type":"ContainerDied","Data":"488c6f214f08e6054c4ba42e5a26c69b65097a47fc857be0e6cbfc804001e125"}
Sep 16 21:28:53.956166 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:28:53.956026    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="488c6f214f08e6054c4ba42e5a26c69b65097a47fc857be0e6cbfc804001e125"
Sep 16 21:29:00.967467 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967436    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"]
Sep 16 21:29:00.967822 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967774    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7423c65-e590-496c-a39c-a053f36c0f80" containerName="util"
Sep 16 21:29:00.967822 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967785    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7423c65-e590-496c-a39c-a053f36c0f80" containerName="util"
Sep 16 21:29:00.967822 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967794    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c50f102-4948-4806-b084-6c63693493fd" containerName="pull"
Sep 16 21:29:00.967822 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967800    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c50f102-4948-4806-b084-6c63693493fd" containerName="pull"
Sep 16 21:29:00.967822 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967806    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c50f102-4948-4806-b084-6c63693493fd" containerName="extract"
Sep 16 21:29:00.967822 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967812    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c50f102-4948-4806-b084-6c63693493fd" containerName="extract"
Sep 16 21:29:00.967822 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967819    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df4e8739-6f71-4af4-a728-4f3e6a85a9df" containerName="pull"
Sep 16 21:29:00.967822 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967824    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="df4e8739-6f71-4af4-a728-4f3e6a85a9df" containerName="pull"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967832    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1440fda4-9c7e-4740-a5f6-e9bc20dbece8" containerName="pull"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967838    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1440fda4-9c7e-4740-a5f6-e9bc20dbece8" containerName="pull"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967850    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c50f102-4948-4806-b084-6c63693493fd" containerName="util"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967855    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c50f102-4948-4806-b084-6c63693493fd" containerName="util"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967861    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1440fda4-9c7e-4740-a5f6-e9bc20dbece8" containerName="extract"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967867    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1440fda4-9c7e-4740-a5f6-e9bc20dbece8" containerName="extract"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967874    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1440fda4-9c7e-4740-a5f6-e9bc20dbece8" containerName="util"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967878    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1440fda4-9c7e-4740-a5f6-e9bc20dbece8" containerName="util"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967885    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df4e8739-6f71-4af4-a728-4f3e6a85a9df" containerName="extract"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967889    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="df4e8739-6f71-4af4-a728-4f3e6a85a9df" containerName="extract"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967897    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7423c65-e590-496c-a39c-a053f36c0f80" containerName="extract"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967902    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7423c65-e590-496c-a39c-a053f36c0f80" containerName="extract"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967909    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df4e8739-6f71-4af4-a728-4f3e6a85a9df" containerName="util"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967913    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="df4e8739-6f71-4af4-a728-4f3e6a85a9df" containerName="util"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967919    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7423c65-e590-496c-a39c-a053f36c0f80" containerName="pull"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967923    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7423c65-e590-496c-a39c-a053f36c0f80" containerName="pull"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967983    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1440fda4-9c7e-4740-a5f6-e9bc20dbece8" containerName="extract"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967990    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7423c65-e590-496c-a39c-a053f36c0f80" containerName="extract"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.967994    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c50f102-4948-4806-b084-6c63693493fd" containerName="extract"
Sep 16 21:29:00.968061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.968001    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="df4e8739-6f71-4af4-a728-4f3e6a85a9df" containerName="extract"
Sep 16 21:29:00.972151 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.972133    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:00.976597 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.976574    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 21:29:00.976732 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.976666    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kgfv7\""
Sep 16 21:29:00.976893 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.976872    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 21:29:00.989683 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:00.989654    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"]
Sep 16 21:29:01.109798 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:01.109762    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08dd59ba-9ad6-4841-92d2-2785f7611879-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" (UID: \"08dd59ba-9ad6-4841-92d2-2785f7611879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:01.110005 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:01.109854    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672sr\" (UniqueName: \"kubernetes.io/projected/08dd59ba-9ad6-4841-92d2-2785f7611879-kube-api-access-672sr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" (UID: \"08dd59ba-9ad6-4841-92d2-2785f7611879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:01.210699 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:01.210666    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08dd59ba-9ad6-4841-92d2-2785f7611879-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" (UID: \"08dd59ba-9ad6-4841-92d2-2785f7611879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:01.210886 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:01.210729    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672sr\" (UniqueName: \"kubernetes.io/projected/08dd59ba-9ad6-4841-92d2-2785f7611879-kube-api-access-672sr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" (UID: \"08dd59ba-9ad6-4841-92d2-2785f7611879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:01.211116 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:01.211079    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08dd59ba-9ad6-4841-92d2-2785f7611879-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" (UID: \"08dd59ba-9ad6-4841-92d2-2785f7611879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:01.226983 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:01.226923    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-672sr\" (UniqueName: \"kubernetes.io/projected/08dd59ba-9ad6-4841-92d2-2785f7611879-kube-api-access-672sr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" (UID: \"08dd59ba-9ad6-4841-92d2-2785f7611879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:01.282583 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:01.282554    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:01.569433 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:01.569347    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"]
Sep 16 21:29:01.572852 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:29:01.572821    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08dd59ba_9ad6_4841_92d2_2785f7611879.slice/crio-7babac8a363402cdf7c7125001a55dfcfc9ffce5ddd5cdabc3eafd46832032ed WatchSource:0}: Error finding container 7babac8a363402cdf7c7125001a55dfcfc9ffce5ddd5cdabc3eafd46832032ed: Status 404 returned error can't find the container with id 7babac8a363402cdf7c7125001a55dfcfc9ffce5ddd5cdabc3eafd46832032ed
Sep 16 21:29:01.998048 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:01.998010    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml" event={"ID":"08dd59ba-9ad6-4841-92d2-2785f7611879","Type":"ContainerStarted","Data":"7babac8a363402cdf7c7125001a55dfcfc9ffce5ddd5cdabc3eafd46832032ed"}
Sep 16 21:29:05.789828 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:05.789793    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz"]
Sep 16 21:29:05.793585 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:05.793563    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz"
Sep 16 21:29:05.795818 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:05.795794    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 21:29:05.795935 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:05.795883    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zsjpb\""
Sep 16 21:29:05.812369 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:05.812346    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz"]
Sep 16 21:29:05.955162 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:05.955098    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gspk\" (UniqueName: \"kubernetes.io/projected/e394fbd6-59c8-4157-8bb5-0fc3e40a70ac-kube-api-access-9gspk\") pod \"dns-operator-controller-manager-7b84cd7c9c-fpstz\" (UID: \"e394fbd6-59c8-4157-8bb5-0fc3e40a70ac\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz"
Sep 16 21:29:06.055760 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:06.055671    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gspk\" (UniqueName: \"kubernetes.io/projected/e394fbd6-59c8-4157-8bb5-0fc3e40a70ac-kube-api-access-9gspk\") pod \"dns-operator-controller-manager-7b84cd7c9c-fpstz\" (UID: \"e394fbd6-59c8-4157-8bb5-0fc3e40a70ac\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz"
Sep 16 21:29:06.065193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:06.065165    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gspk\" (UniqueName: \"kubernetes.io/projected/e394fbd6-59c8-4157-8bb5-0fc3e40a70ac-kube-api-access-9gspk\") pod \"dns-operator-controller-manager-7b84cd7c9c-fpstz\" (UID: \"e394fbd6-59c8-4157-8bb5-0fc3e40a70ac\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz"
Sep 16 21:29:06.105199 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:06.105166    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz"
Sep 16 21:29:06.884697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:06.884675    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz"]
Sep 16 21:29:06.886278 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:29:06.886250    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode394fbd6_59c8_4157_8bb5_0fc3e40a70ac.slice/crio-df0523c9abe8633ec3b3d5e0e8edf867af86f04f2f5a2ec261ef7296e2f20a5a WatchSource:0}: Error finding container df0523c9abe8633ec3b3d5e0e8edf867af86f04f2f5a2ec261ef7296e2f20a5a: Status 404 returned error can't find the container with id df0523c9abe8633ec3b3d5e0e8edf867af86f04f2f5a2ec261ef7296e2f20a5a
Sep 16 21:29:07.030020 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:07.029979    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz" event={"ID":"e394fbd6-59c8-4157-8bb5-0fc3e40a70ac","Type":"ContainerStarted","Data":"df0523c9abe8633ec3b3d5e0e8edf867af86f04f2f5a2ec261ef7296e2f20a5a"}
Sep 16 21:29:07.031777 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:07.031752    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml" event={"ID":"08dd59ba-9ad6-4841-92d2-2785f7611879","Type":"ContainerStarted","Data":"1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1"}
Sep 16 21:29:07.031901 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:07.031823    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:07.053669 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:07.053624    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml" podStartSLOduration=1.936142044 podStartE2EDuration="7.053609975s" totalImagesPullingTime="5.117467931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:07.051238203 +0000 UTC m=+386.436921298" watchObservedRunningTime="2026-09-16 21:29:07.053609975 +0000 UTC m=+386.439293053"
Sep 16 21:29:07.820207 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:07.820174    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"]
Sep 16 21:29:07.823761 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:07.823738    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:07.826185 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:07.826164    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5vk98\""
Sep 16 21:29:07.840466 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:07.840443    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"]
Sep 16 21:29:07.973140 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:07.973102    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l447t\" (UniqueName: \"kubernetes.io/projected/228520f2-befa-4f53-a5f0-d01ba8ab2e03-kube-api-access-l447t\") pod \"limitador-operator-controller-manager-74495c69f8-mcl85\" (UID: \"228520f2-befa-4f53-a5f0-d01ba8ab2e03\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:08.073935 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:08.073864    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l447t\" (UniqueName: \"kubernetes.io/projected/228520f2-befa-4f53-a5f0-d01ba8ab2e03-kube-api-access-l447t\") pod \"limitador-operator-controller-manager-74495c69f8-mcl85\" (UID: \"228520f2-befa-4f53-a5f0-d01ba8ab2e03\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:08.088642 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:08.088615    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l447t\" (UniqueName: \"kubernetes.io/projected/228520f2-befa-4f53-a5f0-d01ba8ab2e03-kube-api-access-l447t\") pod \"limitador-operator-controller-manager-74495c69f8-mcl85\" (UID: \"228520f2-befa-4f53-a5f0-d01ba8ab2e03\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:08.135599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:08.135571    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:08.421143 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:08.421115    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"]
Sep 16 21:29:08.670357 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:29:08.670284    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228520f2_befa_4f53_a5f0_d01ba8ab2e03.slice/crio-0523829bd3d9f282fadc1b295b764785e4342e14139364814995962fcce1cad6 WatchSource:0}: Error finding container 0523829bd3d9f282fadc1b295b764785e4342e14139364814995962fcce1cad6: Status 404 returned error can't find the container with id 0523829bd3d9f282fadc1b295b764785e4342e14139364814995962fcce1cad6
Sep 16 21:29:09.044949 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:09.044910    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85" event={"ID":"228520f2-befa-4f53-a5f0-d01ba8ab2e03","Type":"ContainerStarted","Data":"0523829bd3d9f282fadc1b295b764785e4342e14139364814995962fcce1cad6"}
Sep 16 21:29:09.046609 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:09.046582    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz" event={"ID":"e394fbd6-59c8-4157-8bb5-0fc3e40a70ac","Type":"ContainerStarted","Data":"af1a1dc008b640c94a7061e3a02cb0cc0ac2cc6dbc89c82365dd583e19b14f06"}
Sep 16 21:29:09.046744 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:09.046701    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz"
Sep 16 21:29:09.066857 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:09.066816    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz" podStartSLOduration=2.225735125 podStartE2EDuration="4.066801631s" totalImagesPullingTime="1.841066506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:09.065090045 +0000 UTC m=+388.450773144" watchObservedRunningTime="2026-09-16 21:29:09.066801631 +0000 UTC m=+388.452484714"
Sep 16 21:29:11.060768 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:11.060681    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85" event={"ID":"228520f2-befa-4f53-a5f0-d01ba8ab2e03","Type":"ContainerStarted","Data":"505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6"}
Sep 16 21:29:11.060768 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:11.060758    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:11.079067 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:11.079019    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85" podStartSLOduration=1.961424877 podStartE2EDuration="4.079000576s" totalImagesPullingTime="2.117575699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:11.078221959 +0000 UTC m=+390.463905061" watchObservedRunningTime="2026-09-16 21:29:11.079000576 +0000 UTC m=+390.464683660"
Sep 16 21:29:18.040144 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:18.040112    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:20.025047 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.025018    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"]
Sep 16 21:29:20.028679 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.028660    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:20.051639 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.051611    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"]
Sep 16 21:29:20.056485 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.056465    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fpstz"
Sep 16 21:29:20.070185 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.070162    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxcl\" (UniqueName: \"kubernetes.io/projected/c9d82758-a301-4081-8e7a-a212f1bc7374-kube-api-access-drxcl\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wmkd7\" (UID: \"c9d82758-a301-4081-8e7a-a212f1bc7374\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:20.070313 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.070228    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9d82758-a301-4081-8e7a-a212f1bc7374-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wmkd7\" (UID: \"c9d82758-a301-4081-8e7a-a212f1bc7374\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:20.170721 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.170693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxcl\" (UniqueName: \"kubernetes.io/projected/c9d82758-a301-4081-8e7a-a212f1bc7374-kube-api-access-drxcl\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wmkd7\" (UID: \"c9d82758-a301-4081-8e7a-a212f1bc7374\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:20.170865 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.170793    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9d82758-a301-4081-8e7a-a212f1bc7374-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wmkd7\" (UID: \"c9d82758-a301-4081-8e7a-a212f1bc7374\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:20.171164 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.171145    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9d82758-a301-4081-8e7a-a212f1bc7374-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wmkd7\" (UID: \"c9d82758-a301-4081-8e7a-a212f1bc7374\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:20.176898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.176873    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"]
Sep 16 21:29:20.177054 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.177035    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml" podUID="08dd59ba-9ad6-4841-92d2-2785f7611879" containerName="manager" containerID="cri-o://1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1" gracePeriod=2
Sep 16 21:29:20.186903 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.186697    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"]
Sep 16 21:29:20.189348 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.189329    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxcl\" (UniqueName: \"kubernetes.io/projected/c9d82758-a301-4081-8e7a-a212f1bc7374-kube-api-access-drxcl\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wmkd7\" (UID: \"c9d82758-a301-4081-8e7a-a212f1bc7374\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:20.203151 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.203114    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"]
Sep 16 21:29:20.204646 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.203498    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:20.227257 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.227230    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"]
Sep 16 21:29:20.229312 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.229291    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"]
Sep 16 21:29:20.229654 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.229637    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08dd59ba-9ad6-4841-92d2-2785f7611879" containerName="manager"
Sep 16 21:29:20.229654 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.229654    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="08dd59ba-9ad6-4841-92d2-2785f7611879" containerName="manager"
Sep 16 21:29:20.229777 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.229718    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="08dd59ba-9ad6-4841-92d2-2785f7611879" containerName="manager"
Sep 16 21:29:20.233694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.233678    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:20.235808 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:20.235775    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="08dd59ba-9ad6-4841-92d2-2785f7611879" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:20.241982 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.241961    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"]
Sep 16 21:29:20.242151 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.242129    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85" podUID="228520f2-befa-4f53-a5f0-d01ba8ab2e03" containerName="manager" containerID="cri-o://505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6" gracePeriod=2
Sep 16 21:29:20.243904 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.243883    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:20.264290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.264265    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"]
Sep 16 21:29:20.270057 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.270039    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"]
Sep 16 21:29:20.270149 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.270121    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"
Sep 16 21:29:20.270579 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.270546    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"]
Sep 16 21:29:20.271370 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.271343    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-j5kpb\" (UID: \"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:20.271517 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.271427    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgrcz\" (UniqueName: \"kubernetes.io/projected/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-kube-api-access-xgrcz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-j5kpb\" (UID: \"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:20.273712 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.273690    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"]
Sep 16 21:29:20.277218 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.277173    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77"]
Sep 16 21:29:20.277528 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.277515    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228520f2-befa-4f53-a5f0-d01ba8ab2e03" containerName="manager"
Sep 16 21:29:20.277577 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.277531    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="228520f2-befa-4f53-a5f0-d01ba8ab2e03" containerName="manager"
Sep 16 21:29:20.277619 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.277591    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="228520f2-befa-4f53-a5f0-d01ba8ab2e03" containerName="manager"
Sep 16 21:29:20.280378 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.280350    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77"
Sep 16 21:29:20.280536 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:20.280506    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mcl85\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="228520f2-befa-4f53-a5f0-d01ba8ab2e03" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:20.282150 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:20.282126    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="08dd59ba-9ad6-4841-92d2-2785f7611879" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:20.294266 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.294245    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"]
Sep 16 21:29:20.301887 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.301864    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77"]
Sep 16 21:29:20.313170 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.313144    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"]
Sep 16 21:29:20.316189 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:20.316151    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-pl67r], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-pl67r]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w" podUID="f676b0b5-4bc9-45ec-802f-529fab5297ef"
Sep 16 21:29:20.317191 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.317174    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:29:20.346812 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:20.346766    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mcl85\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="228520f2-befa-4f53-a5f0-d01ba8ab2e03" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:20.348737 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.348711    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"]
Sep 16 21:29:20.372761 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.372728    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f676b0b5-4bc9-45ec-802f-529fab5297ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8648w\" (UID: \"f676b0b5-4bc9-45ec-802f-529fab5297ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"
Sep 16 21:29:20.372761 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.372767    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tth9p\" (UniqueName: \"kubernetes.io/projected/1557051d-7d49-44a5-8811-664cd89a72a2-kube-api-access-tth9p\") pod \"limitador-operator-controller-manager-74495c69f8-48k77\" (UID: \"1557051d-7d49-44a5-8811-664cd89a72a2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77"
Sep 16 21:29:20.372996 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.372814    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5tc\" (UniqueName: \"kubernetes.io/projected/82aba2fe-2314-44ec-a8a8-404b6db3af87-kube-api-access-jb5tc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tnvgm\" (UID: \"82aba2fe-2314-44ec-a8a8-404b6db3af87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:29:20.372996 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.372834    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82aba2fe-2314-44ec-a8a8-404b6db3af87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tnvgm\" (UID: \"82aba2fe-2314-44ec-a8a8-404b6db3af87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:29:20.372996 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.372875    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-j5kpb\" (UID: \"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:20.372996 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.372952    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl67r\" (UniqueName: \"kubernetes.io/projected/f676b0b5-4bc9-45ec-802f-529fab5297ef-kube-api-access-pl67r\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8648w\" (UID: \"f676b0b5-4bc9-45ec-802f-529fab5297ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"
Sep 16 21:29:20.373216 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.372998    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrcz\" (UniqueName: \"kubernetes.io/projected/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-kube-api-access-xgrcz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-j5kpb\" (UID: \"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:20.373281 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.373212    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-j5kpb\" (UID: \"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:20.378471 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:20.378435    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="08dd59ba-9ad6-4841-92d2-2785f7611879" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:20.390630 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.390602    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrcz\" (UniqueName: \"kubernetes.io/projected/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-kube-api-access-xgrcz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-j5kpb\" (UID: \"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:20.472950 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.472924    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:20.473456 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.473434    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5tc\" (UniqueName: \"kubernetes.io/projected/82aba2fe-2314-44ec-a8a8-404b6db3af87-kube-api-access-jb5tc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tnvgm\" (UID: \"82aba2fe-2314-44ec-a8a8-404b6db3af87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:29:20.473542 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.473468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82aba2fe-2314-44ec-a8a8-404b6db3af87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tnvgm\" (UID: \"82aba2fe-2314-44ec-a8a8-404b6db3af87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:29:20.473608 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.473590    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl67r\" (UniqueName: \"kubernetes.io/projected/f676b0b5-4bc9-45ec-802f-529fab5297ef-kube-api-access-pl67r\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8648w\" (UID: \"f676b0b5-4bc9-45ec-802f-529fab5297ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"
Sep 16 21:29:20.473692 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.473677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f676b0b5-4bc9-45ec-802f-529fab5297ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8648w\" (UID: \"f676b0b5-4bc9-45ec-802f-529fab5297ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"
Sep 16 21:29:20.473738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.473723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tth9p\" (UniqueName: \"kubernetes.io/projected/1557051d-7d49-44a5-8811-664cd89a72a2-kube-api-access-tth9p\") pod \"limitador-operator-controller-manager-74495c69f8-48k77\" (UID: \"1557051d-7d49-44a5-8811-664cd89a72a2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77"
Sep 16 21:29:20.473815 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.473797    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82aba2fe-2314-44ec-a8a8-404b6db3af87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tnvgm\" (UID: \"82aba2fe-2314-44ec-a8a8-404b6db3af87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:29:20.474024 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.474001    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f676b0b5-4bc9-45ec-802f-529fab5297ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8648w\" (UID: \"f676b0b5-4bc9-45ec-802f-529fab5297ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"
Sep 16 21:29:20.479725 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.479707    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:20.486583 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:20.486555    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mcl85\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="228520f2-befa-4f53-a5f0-d01ba8ab2e03" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:20.488434 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:20.488414    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="08dd59ba-9ad6-4841-92d2-2785f7611879" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:20.490680 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:20.490656    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mcl85\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="228520f2-befa-4f53-a5f0-d01ba8ab2e03" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:20.492872 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:20.492854    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="08dd59ba-9ad6-4841-92d2-2785f7611879" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:20.496157 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.496134    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5tc\" (UniqueName: \"kubernetes.io/projected/82aba2fe-2314-44ec-a8a8-404b6db3af87-kube-api-access-jb5tc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tnvgm\" (UID: \"82aba2fe-2314-44ec-a8a8-404b6db3af87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:29:20.496261 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.496222    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl67r\" (UniqueName: \"kubernetes.io/projected/f676b0b5-4bc9-45ec-802f-529fab5297ef-kube-api-access-pl67r\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8648w\" (UID: \"f676b0b5-4bc9-45ec-802f-529fab5297ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"
Sep 16 21:29:20.496500 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.496480    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tth9p\" (UniqueName: \"kubernetes.io/projected/1557051d-7d49-44a5-8811-664cd89a72a2-kube-api-access-tth9p\") pod \"limitador-operator-controller-manager-74495c69f8-48k77\" (UID: \"1557051d-7d49-44a5-8811-664cd89a72a2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77"
Sep 16 21:29:20.574178 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.574119    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08dd59ba-9ad6-4841-92d2-2785f7611879-extensions-socket-volume\") pod \"08dd59ba-9ad6-4841-92d2-2785f7611879\" (UID: \"08dd59ba-9ad6-4841-92d2-2785f7611879\") "
Sep 16 21:29:20.574178 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.574154    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-672sr\" (UniqueName: \"kubernetes.io/projected/08dd59ba-9ad6-4841-92d2-2785f7611879-kube-api-access-672sr\") pod \"08dd59ba-9ad6-4841-92d2-2785f7611879\" (UID: \"08dd59ba-9ad6-4841-92d2-2785f7611879\") "
Sep 16 21:29:20.574305 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.574228    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l447t\" (UniqueName: \"kubernetes.io/projected/228520f2-befa-4f53-a5f0-d01ba8ab2e03-kube-api-access-l447t\") pod \"228520f2-befa-4f53-a5f0-d01ba8ab2e03\" (UID: \"228520f2-befa-4f53-a5f0-d01ba8ab2e03\") "
Sep 16 21:29:20.574676 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.574655    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08dd59ba-9ad6-4841-92d2-2785f7611879-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "08dd59ba-9ad6-4841-92d2-2785f7611879" (UID: "08dd59ba-9ad6-4841-92d2-2785f7611879"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:29:20.576179 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.576162    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08dd59ba-9ad6-4841-92d2-2785f7611879-kube-api-access-672sr" (OuterVolumeSpecName: "kube-api-access-672sr") pod "08dd59ba-9ad6-4841-92d2-2785f7611879" (UID: "08dd59ba-9ad6-4841-92d2-2785f7611879"). InnerVolumeSpecName "kube-api-access-672sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:29:20.576243 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.576203    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228520f2-befa-4f53-a5f0-d01ba8ab2e03-kube-api-access-l447t" (OuterVolumeSpecName: "kube-api-access-l447t") pod "228520f2-befa-4f53-a5f0-d01ba8ab2e03" (UID: "228520f2-befa-4f53-a5f0-d01ba8ab2e03"). InnerVolumeSpecName "kube-api-access-l447t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:29:20.608366 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.608343    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:20.623513 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.623492    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77"
Sep 16 21:29:20.629437 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.629416    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:29:20.674976 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.674872    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l447t\" (UniqueName: \"kubernetes.io/projected/228520f2-befa-4f53-a5f0-d01ba8ab2e03-kube-api-access-l447t\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:29:20.674976 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.674911    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08dd59ba-9ad6-4841-92d2-2785f7611879-extensions-socket-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:29:20.674976 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.674927    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-672sr\" (UniqueName: \"kubernetes.io/projected/08dd59ba-9ad6-4841-92d2-2785f7611879-kube-api-access-672sr\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:29:20.916381 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.916308    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"]
Sep 16 21:29:20.920571 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:29:20.920544    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20b3f4c_16f7_4ed6_9d2a_a0dc2166a918.slice/crio-ede299faf9460fe087c8e3d9cdff8b0fe5f678b7386ece93ef613421cbf2b074 WatchSource:0}: Error finding container ede299faf9460fe087c8e3d9cdff8b0fe5f678b7386ece93ef613421cbf2b074: Status 404 returned error can't find the container with id ede299faf9460fe087c8e3d9cdff8b0fe5f678b7386ece93ef613421cbf2b074
Sep 16 21:29:20.927096 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.926894    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77"]
Sep 16 21:29:20.928890 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:29:20.928603    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1557051d_7d49_44a5_8811_664cd89a72a2.slice/crio-75413bec4789028593152b6afa436bc5d583194eaa2a4e1e9b90f0f7c2162762 WatchSource:0}: Error finding container 75413bec4789028593152b6afa436bc5d583194eaa2a4e1e9b90f0f7c2162762: Status 404 returned error can't find the container with id 75413bec4789028593152b6afa436bc5d583194eaa2a4e1e9b90f0f7c2162762
Sep 16 21:29:20.930289 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:20.929995    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"]
Sep 16 21:29:20.934934 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:29:20.934913    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82aba2fe_2314_44ec_a8a8_404b6db3af87.slice/crio-e92c5bbbd5d540dab754f9dc90d34103811883d2af696d0efc48ef34ff8eed8c WatchSource:0}: Error finding container e92c5bbbd5d540dab754f9dc90d34103811883d2af696d0efc48ef34ff8eed8c: Status 404 returned error can't find the container with id e92c5bbbd5d540dab754f9dc90d34103811883d2af696d0efc48ef34ff8eed8c
Sep 16 21:29:21.108627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.108586    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08dd59ba-9ad6-4841-92d2-2785f7611879" path="/var/lib/kubelet/pods/08dd59ba-9ad6-4841-92d2-2785f7611879/volumes"
Sep 16 21:29:21.109063 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.109033    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="228520f2-befa-4f53-a5f0-d01ba8ab2e03" path="/var/lib/kubelet/pods/228520f2-befa-4f53-a5f0-d01ba8ab2e03/volumes"
Sep 16 21:29:21.111679 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:21.111647    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mcl85\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="228520f2-befa-4f53-a5f0-d01ba8ab2e03" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:21.113352 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:21.113321    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m64ml\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="08dd59ba-9ad6-4841-92d2-2785f7611879" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:21.115328 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.115304    2567 generic.go:356] "Generic (PLEG): container finished" podID="228520f2-befa-4f53-a5f0-d01ba8ab2e03" containerID="505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6" exitCode=0
Sep 16 21:29:21.115436 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.115348    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcl85"
Sep 16 21:29:21.115436 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.115392    2567 scope.go:117] "RemoveContainer" containerID="505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6"
Sep 16 21:29:21.117616 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.117361    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77" event={"ID":"1557051d-7d49-44a5-8811-664cd89a72a2","Type":"ContainerStarted","Data":"e4965f997a80b3a05bb122bdae4386bf4195c910a6817482126e810a1add03b9"}
Sep 16 21:29:21.117616 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.117429    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77"
Sep 16 21:29:21.117616 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.117457    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77" event={"ID":"1557051d-7d49-44a5-8811-664cd89a72a2","Type":"ContainerStarted","Data":"75413bec4789028593152b6afa436bc5d583194eaa2a4e1e9b90f0f7c2162762"}
Sep 16 21:29:21.119388 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.119344    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm" event={"ID":"82aba2fe-2314-44ec-a8a8-404b6db3af87","Type":"ContainerStarted","Data":"bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb"}
Sep 16 21:29:21.119496 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.119388    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm" event={"ID":"82aba2fe-2314-44ec-a8a8-404b6db3af87","Type":"ContainerStarted","Data":"e92c5bbbd5d540dab754f9dc90d34103811883d2af696d0efc48ef34ff8eed8c"}
Sep 16 21:29:21.119496 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.119448    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:29:21.121172 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.121146    2567 generic.go:356] "Generic (PLEG): container finished" podID="08dd59ba-9ad6-4841-92d2-2785f7611879" containerID="1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1" exitCode=0
Sep 16 21:29:21.121275 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.121189    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m64ml"
Sep 16 21:29:21.124772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.124751    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"
Sep 16 21:29:21.125368 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.125350    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb" event={"ID":"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918","Type":"ContainerStarted","Data":"86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50"}
Sep 16 21:29:21.125465 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.125386    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:21.125465 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.125425    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb" event={"ID":"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918","Type":"ContainerStarted","Data":"ede299faf9460fe087c8e3d9cdff8b0fe5f678b7386ece93ef613421cbf2b074"}
Sep 16 21:29:21.132864 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.131779    2567 scope.go:117] "RemoveContainer" containerID="505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6"
Sep 16 21:29:21.136443 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:21.133265    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6\": container with ID starting with 505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6 not found: ID does not exist" containerID="505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6"
Sep 16 21:29:21.136443 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.133306    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6"} err="failed to get container status \"505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6\": rpc error: code = NotFound desc = could not find container \"505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6\": container with ID starting with 505b1b6d8081de524e19374141ce178eb08a4a6af0a61b53c9ce02eafb971df6 not found: ID does not exist"
Sep 16 21:29:21.136443 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.133327    2567 scope.go:117] "RemoveContainer" containerID="1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1"
Sep 16 21:29:21.143626 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.143608    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"
Sep 16 21:29:21.147505 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.147491    2567 scope.go:117] "RemoveContainer" containerID="1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1"
Sep 16 21:29:21.147750 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:21.147728    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1\": container with ID starting with 1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1 not found: ID does not exist" containerID="1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1"
Sep 16 21:29:21.147791 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.147761    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1"} err="failed to get container status \"1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1\": rpc error: code = NotFound desc = could not find container \"1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1\": container with ID starting with 1e7e761b7c03b0003b5166e553caa6c27f13bde51804af6819f5694e9acd43b1 not found: ID does not exist"
Sep 16 21:29:21.154150 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.154113    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77" podStartSLOduration=1.154100203 podStartE2EDuration="1.154100203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:21.152627516 +0000 UTC m=+400.538310619" watchObservedRunningTime="2026-09-16 21:29:21.154100203 +0000 UTC m=+400.539783288"
Sep 16 21:29:21.179061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.179012    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl67r\" (UniqueName: \"kubernetes.io/projected/f676b0b5-4bc9-45ec-802f-529fab5297ef-kube-api-access-pl67r\") pod \"f676b0b5-4bc9-45ec-802f-529fab5297ef\" (UID: \"f676b0b5-4bc9-45ec-802f-529fab5297ef\") "
Sep 16 21:29:21.179152 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.179072    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f676b0b5-4bc9-45ec-802f-529fab5297ef-extensions-socket-volume\") pod \"f676b0b5-4bc9-45ec-802f-529fab5297ef\" (UID: \"f676b0b5-4bc9-45ec-802f-529fab5297ef\") "
Sep 16 21:29:21.179340 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.179302    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f676b0b5-4bc9-45ec-802f-529fab5297ef-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f676b0b5-4bc9-45ec-802f-529fab5297ef" (UID: "f676b0b5-4bc9-45ec-802f-529fab5297ef"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:29:21.181336 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.181311    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f676b0b5-4bc9-45ec-802f-529fab5297ef-kube-api-access-pl67r" (OuterVolumeSpecName: "kube-api-access-pl67r") pod "f676b0b5-4bc9-45ec-802f-529fab5297ef" (UID: "f676b0b5-4bc9-45ec-802f-529fab5297ef"). InnerVolumeSpecName "kube-api-access-pl67r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:29:21.208050 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.208003    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb" podStartSLOduration=1.207986376 podStartE2EDuration="1.207986376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:21.205743818 +0000 UTC m=+400.591426918" watchObservedRunningTime="2026-09-16 21:29:21.207986376 +0000 UTC m=+400.593669462"
Sep 16 21:29:21.237849 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.237806    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm" podStartSLOduration=1.237787911 podStartE2EDuration="1.237787911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:21.23558199 +0000 UTC m=+400.621265124" watchObservedRunningTime="2026-09-16 21:29:21.237787911 +0000 UTC m=+400.623470995"
Sep 16 21:29:21.280061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.280035    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f676b0b5-4bc9-45ec-802f-529fab5297ef-extensions-socket-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:29:21.280061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:21.280059    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl67r\" (UniqueName: \"kubernetes.io/projected/f676b0b5-4bc9-45ec-802f-529fab5297ef-kube-api-access-pl67r\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:29:22.134077 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:22.134038    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"
Sep 16 21:29:22.173057 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:22.173032    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"]
Sep 16 21:29:22.176581 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:22.176555    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8648w"]
Sep 16 21:29:22.746454 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:22.746419    2567 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-wmkd7_kuadrant-system_c9d82758-a301-4081-8e7a-a212f1bc7374_0(2d8dfda5ede5f10614e2ed8e24f079286c69eaaf98e81268452cf178987d2d78): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-wmkd7 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Sep 16 21:29:22.746548 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:22.746479    2567 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-wmkd7_kuadrant-system_c9d82758-a301-4081-8e7a-a212f1bc7374_0(2d8dfda5ede5f10614e2ed8e24f079286c69eaaf98e81268452cf178987d2d78): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-wmkd7 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:23.109103 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:23.109019    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f676b0b5-4bc9-45ec-802f-529fab5297ef" path="/var/lib/kubelet/pods/f676b0b5-4bc9-45ec-802f-529fab5297ef/volumes"
Sep 16 21:29:23.139550 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:23.139517    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:23.147755 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:23.147735    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:23.149748 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:23.149726    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-wmkd7\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="c9d82758-a301-4081-8e7a-a212f1bc7374" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:23.193107 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:23.193085    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drxcl\" (UniqueName: \"kubernetes.io/projected/c9d82758-a301-4081-8e7a-a212f1bc7374-kube-api-access-drxcl\") pod \"c9d82758-a301-4081-8e7a-a212f1bc7374\" (UID: \"c9d82758-a301-4081-8e7a-a212f1bc7374\") "
Sep 16 21:29:23.193201 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:23.193148    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9d82758-a301-4081-8e7a-a212f1bc7374-extensions-socket-volume\") pod \"c9d82758-a301-4081-8e7a-a212f1bc7374\" (UID: \"c9d82758-a301-4081-8e7a-a212f1bc7374\") "
Sep 16 21:29:23.193441 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:23.193418    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d82758-a301-4081-8e7a-a212f1bc7374-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c9d82758-a301-4081-8e7a-a212f1bc7374" (UID: "c9d82758-a301-4081-8e7a-a212f1bc7374"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:29:23.195141 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:23.195122    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d82758-a301-4081-8e7a-a212f1bc7374-kube-api-access-drxcl" (OuterVolumeSpecName: "kube-api-access-drxcl") pod "c9d82758-a301-4081-8e7a-a212f1bc7374" (UID: "c9d82758-a301-4081-8e7a-a212f1bc7374"). InnerVolumeSpecName "kube-api-access-drxcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:29:23.294473 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:23.294452    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drxcl\" (UniqueName: \"kubernetes.io/projected/c9d82758-a301-4081-8e7a-a212f1bc7374-kube-api-access-drxcl\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:29:23.294473 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:23.294474    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9d82758-a301-4081-8e7a-a212f1bc7374-extensions-socket-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:29:24.145977 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:24.145941    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:24.147800 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:24.147772    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-wmkd7\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="c9d82758-a301-4081-8e7a-a212f1bc7374" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:24.153260 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:24.153227    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-wmkd7\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="c9d82758-a301-4081-8e7a-a212f1bc7374" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wmkd7"
Sep 16 21:29:25.108578 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:25.108543    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9d82758-a301-4081-8e7a-a212f1bc7374" path="/var/lib/kubelet/pods/c9d82758-a301-4081-8e7a-a212f1bc7374/volumes"
Sep 16 21:29:32.135943 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.135905    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-48k77"
Sep 16 21:29:32.136335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.136083    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:32.136335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.136136    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:29:32.226509 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.226470    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"]
Sep 16 21:29:32.226762 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.226732    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb" podUID="f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918" containerName="manager" containerID="cri-o://86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50" gracePeriod=10
Sep 16 21:29:32.465068 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.465041    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:32.564958 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.564930    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-extensions-socket-volume\") pod \"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918\" (UID: \"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918\") "
Sep 16 21:29:32.565111 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.565039    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgrcz\" (UniqueName: \"kubernetes.io/projected/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-kube-api-access-xgrcz\") pod \"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918\" (UID: \"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918\") "
Sep 16 21:29:32.565307 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.565283    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918" (UID: "f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:29:32.566848 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.566826    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-kube-api-access-xgrcz" (OuterVolumeSpecName: "kube-api-access-xgrcz") pod "f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918" (UID: "f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918"). InnerVolumeSpecName "kube-api-access-xgrcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:29:32.666104 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.666078    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgrcz\" (UniqueName: \"kubernetes.io/projected/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-kube-api-access-xgrcz\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:29:32.666104 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:32.666099    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918-extensions-socket-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:29:33.200089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:33.200052    2567 generic.go:356] "Generic (PLEG): container finished" podID="f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918" containerID="86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50" exitCode=0
Sep 16 21:29:33.200516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:33.200098    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"
Sep 16 21:29:33.200516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:33.200135    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb" event={"ID":"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918","Type":"ContainerDied","Data":"86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50"}
Sep 16 21:29:33.200516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:33.200190    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb" event={"ID":"f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918","Type":"ContainerDied","Data":"ede299faf9460fe087c8e3d9cdff8b0fe5f678b7386ece93ef613421cbf2b074"}
Sep 16 21:29:33.200516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:33.200215    2567 scope.go:117] "RemoveContainer" containerID="86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50"
Sep 16 21:29:33.208983 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:33.208968    2567 scope.go:117] "RemoveContainer" containerID="86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50"
Sep 16 21:29:33.209293 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:29:33.209266    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50\": container with ID starting with 86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50 not found: ID does not exist" containerID="86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50"
Sep 16 21:29:33.209342 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:33.209302    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50"} err="failed to get container status \"86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50\": rpc error: code = NotFound desc = could not find container \"86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50\": container with ID starting with 86a09c7346cdf29af2cd176cf79ff42000082a3a490b7b1ee629398559757b50 not found: ID does not exist"
Sep 16 21:29:33.224321 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:33.224294    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"]
Sep 16 21:29:33.235664 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:33.235643    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-j5kpb"]
Sep 16 21:29:35.108171 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:35.108135    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918" path="/var/lib/kubelet/pods/f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918/volumes"
Sep 16 21:29:51.237267 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.237226    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:29:51.237795 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.237772    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918" containerName="manager"
Sep 16 21:29:51.237845 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.237798    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918" containerName="manager"
Sep 16 21:29:51.237918 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.237907    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20b3f4c-16f7-4ed6-9d2a-a0dc2166a918" containerName="manager"
Sep 16 21:29:51.244898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.244880    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4"
Sep 16 21:29:51.247467 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.247446    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 21:29:51.247649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.247636    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hkhcz\""
Sep 16 21:29:51.251828 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.251807    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:29:51.289890 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.289843    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:29:51.312719 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.312693    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2bacf858-59b2-48be-81e4-b2eb98d0b66f-config-file\") pod \"limitador-limitador-84994f7ffc-zljw4\" (UID: \"2bacf858-59b2-48be-81e4-b2eb98d0b66f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4"
Sep 16 21:29:51.312823 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.312728    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cmxf\" (UniqueName: \"kubernetes.io/projected/2bacf858-59b2-48be-81e4-b2eb98d0b66f-kube-api-access-5cmxf\") pod \"limitador-limitador-84994f7ffc-zljw4\" (UID: \"2bacf858-59b2-48be-81e4-b2eb98d0b66f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4"
Sep 16 21:29:51.413323 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.413295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2bacf858-59b2-48be-81e4-b2eb98d0b66f-config-file\") pod \"limitador-limitador-84994f7ffc-zljw4\" (UID: \"2bacf858-59b2-48be-81e4-b2eb98d0b66f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4"
Sep 16 21:29:51.413457 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.413330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cmxf\" (UniqueName: \"kubernetes.io/projected/2bacf858-59b2-48be-81e4-b2eb98d0b66f-kube-api-access-5cmxf\") pod \"limitador-limitador-84994f7ffc-zljw4\" (UID: \"2bacf858-59b2-48be-81e4-b2eb98d0b66f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4"
Sep 16 21:29:51.413895 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.413874    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2bacf858-59b2-48be-81e4-b2eb98d0b66f-config-file\") pod \"limitador-limitador-84994f7ffc-zljw4\" (UID: \"2bacf858-59b2-48be-81e4-b2eb98d0b66f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4"
Sep 16 21:29:51.421342 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.421317    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cmxf\" (UniqueName: \"kubernetes.io/projected/2bacf858-59b2-48be-81e4-b2eb98d0b66f-kube-api-access-5cmxf\") pod \"limitador-limitador-84994f7ffc-zljw4\" (UID: \"2bacf858-59b2-48be-81e4-b2eb98d0b66f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4"
Sep 16 21:29:51.556333 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.556280    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4"
Sep 16 21:29:51.828910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:51.828885    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:29:51.830746 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:29:51.830710    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bacf858_59b2_48be_81e4_b2eb98d0b66f.slice/crio-3c3a37c94215015debf6498a75999611c518cfe6e96ba63d7681daadea2d5e87 WatchSource:0}: Error finding container 3c3a37c94215015debf6498a75999611c518cfe6e96ba63d7681daadea2d5e87: Status 404 returned error can't find the container with id 3c3a37c94215015debf6498a75999611c518cfe6e96ba63d7681daadea2d5e87
Sep 16 21:29:52.307791 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:52.307757    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4" event={"ID":"2bacf858-59b2-48be-81e4-b2eb98d0b66f","Type":"ContainerStarted","Data":"3c3a37c94215015debf6498a75999611c518cfe6e96ba63d7681daadea2d5e87"}
Sep 16 21:29:53.314275 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:53.314240    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4" event={"ID":"2bacf858-59b2-48be-81e4-b2eb98d0b66f","Type":"ContainerStarted","Data":"7fb1c45ffac5e8b5dea39961768298efc3a1d9f3f728d9e6e55e6182fe42eaff"}
Sep 16 21:29:53.314721 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:53.314367    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4"
Sep 16 21:29:53.332173 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:29:53.332129    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4" podStartSLOduration=1.048764814 podStartE2EDuration="2.332116634s" totalImagesPullingTime="1.28335182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:53.329513907 +0000 UTC m=+432.715197015" watchObservedRunningTime="2026-09-16 21:29:53.332116634 +0000 UTC m=+432.717799718"
Sep 16 21:30:04.321697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:30:04.321617    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-zljw4"
Sep 16 21:31:03.493502 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.493461    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:31:03.496931 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.496905    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-px5dq"]
Sep 16 21:31:03.500269 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.500250    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq"
Sep 16 21:31:03.502571 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.502550    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nrlmm\""
Sep 16 21:31:03.502673 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.502585    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 21:31:03.511551 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.511529    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-px5dq"]
Sep 16 21:31:03.596214 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.596188    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2babf7f2-b5ca-4aa7-af2a-bd60178bac8c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-px5dq\" (UID: \"2babf7f2-b5ca-4aa7-af2a-bd60178bac8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq"
Sep 16 21:31:03.596356 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.596243    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24sq\" (UniqueName: \"kubernetes.io/projected/2babf7f2-b5ca-4aa7-af2a-bd60178bac8c-kube-api-access-l24sq\") pod \"payload-processing-5b54b7fbb7-px5dq\" (UID: \"2babf7f2-b5ca-4aa7-af2a-bd60178bac8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq"
Sep 16 21:31:03.696824 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.696789    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24sq\" (UniqueName: \"kubernetes.io/projected/2babf7f2-b5ca-4aa7-af2a-bd60178bac8c-kube-api-access-l24sq\") pod \"payload-processing-5b54b7fbb7-px5dq\" (UID: \"2babf7f2-b5ca-4aa7-af2a-bd60178bac8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq"
Sep 16 21:31:03.696982 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.696875    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2babf7f2-b5ca-4aa7-af2a-bd60178bac8c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-px5dq\" (UID: \"2babf7f2-b5ca-4aa7-af2a-bd60178bac8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq"
Sep 16 21:31:03.697577 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.697555    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2babf7f2-b5ca-4aa7-af2a-bd60178bac8c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-px5dq\" (UID: \"2babf7f2-b5ca-4aa7-af2a-bd60178bac8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq"
Sep 16 21:31:03.704561 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.704536    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24sq\" (UniqueName: \"kubernetes.io/projected/2babf7f2-b5ca-4aa7-af2a-bd60178bac8c-kube-api-access-l24sq\") pod \"payload-processing-5b54b7fbb7-px5dq\" (UID: \"2babf7f2-b5ca-4aa7-af2a-bd60178bac8c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq"
Sep 16 21:31:03.811059 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:03.810996    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq"
Sep 16 21:31:04.082973 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:04.082951    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-px5dq"]
Sep 16 21:31:04.084666 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:31:04.084629    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2babf7f2_b5ca_4aa7_af2a_bd60178bac8c.slice/crio-79ee432539879d819f580a15e420e4855bc9d0da20ea034b32d4d9201f2d7ab8 WatchSource:0}: Error finding container 79ee432539879d819f580a15e420e4855bc9d0da20ea034b32d4d9201f2d7ab8: Status 404 returned error can't find the container with id 79ee432539879d819f580a15e420e4855bc9d0da20ea034b32d4d9201f2d7ab8
Sep 16 21:31:04.687795 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:04.687756    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq" event={"ID":"2babf7f2-b5ca-4aa7-af2a-bd60178bac8c","Type":"ContainerStarted","Data":"79ee432539879d819f580a15e420e4855bc9d0da20ea034b32d4d9201f2d7ab8"}
Sep 16 21:31:05.694632 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:05.694591    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq" event={"ID":"2babf7f2-b5ca-4aa7-af2a-bd60178bac8c","Type":"ContainerStarted","Data":"1e66c5f78e453c54be627c7dbfc0bd0ae559f048dacef050e1d5de2a3675b65f"}
Sep 16 21:31:05.695022 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:05.694662    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq"
Sep 16 21:31:05.711426 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:05.711358    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq" podStartSLOduration=1.2990156 podStartE2EDuration="2.711339362s" totalImagesPullingTime="1.412323762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:31:05.708513406 +0000 UTC m=+505.094196503" watchObservedRunningTime="2026-09-16 21:31:05.711339362 +0000 UTC m=+505.097022447"
Sep 16 21:31:16.701699 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:16.701670    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-px5dq"
Sep 16 21:31:23.398872 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:23.398833    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:31:31.489739 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:31.489701    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:31:52.181042 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:52.180970    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:31:57.980125 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:57.980089    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"]
Sep 16 21:31:57.984712 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:57.984689    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:57.987500 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:57.987479    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pjb85\""
Sep 16 21:31:57.987697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:57.987508    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 16 21:31:57.987697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:57.987581    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 21:31:57.987697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:57.987584    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 21:31:57.991742 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:57.991718    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"]
Sep 16 21:31:58.140541 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.140509    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.140541 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.140545    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.140735 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.140569    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.140735 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.140591    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5fm\" (UniqueName: \"kubernetes.io/projected/af2d1205-80da-4300-8db9-f5aa457d74e8-kube-api-access-lh5fm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.140735 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.140640    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.140735 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.140698    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af2d1205-80da-4300-8db9-f5aa457d74e8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.242532 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.242428    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.242532 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.242497    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.242740 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.242542    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.242858 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.242842    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5fm\" (UniqueName: \"kubernetes.io/projected/af2d1205-80da-4300-8db9-f5aa457d74e8-kube-api-access-lh5fm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.242902 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.242873    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.242902 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.242797    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.242986 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.242905    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af2d1205-80da-4300-8db9-f5aa457d74e8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.243039 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.243020    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.243296 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.243276    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.245075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.245048    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af2d1205-80da-4300-8db9-f5aa457d74e8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.245612 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.245593    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af2d1205-80da-4300-8db9-f5aa457d74e8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.250175 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.250150    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5fm\" (UniqueName: \"kubernetes.io/projected/af2d1205-80da-4300-8db9-f5aa457d74e8-kube-api-access-lh5fm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p4qwx\" (UID: \"af2d1205-80da-4300-8db9-f5aa457d74e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.295646 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.295622    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:31:58.571884 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.571861    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"]
Sep 16 21:31:58.573899 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:31:58.573861    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2d1205_80da_4300_8db9_f5aa457d74e8.slice/crio-ca1a8c9c3b8f380e29e70ce879530ce11068583368d40ad4983caf0c6a1890f5 WatchSource:0}: Error finding container ca1a8c9c3b8f380e29e70ce879530ce11068583368d40ad4983caf0c6a1890f5: Status 404 returned error can't find the container with id ca1a8c9c3b8f380e29e70ce879530ce11068583368d40ad4983caf0c6a1890f5
Sep 16 21:31:58.989649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:58.989611    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx" event={"ID":"af2d1205-80da-4300-8db9-f5aa457d74e8","Type":"ContainerStarted","Data":"ca1a8c9c3b8f380e29e70ce879530ce11068583368d40ad4983caf0c6a1890f5"}
Sep 16 21:31:59.588160 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:59.588131    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:31:59.996858 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:31:59.996824    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx" event={"ID":"af2d1205-80da-4300-8db9-f5aa457d74e8","Type":"ContainerStarted","Data":"cb86e8742adb27ce8423733e152b2d338fbfbbc0ae83dd88540c9b76eab63c3e"}
Sep 16 21:32:00.016570 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:32:00.016520    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx" podStartSLOduration=2.040017921 podStartE2EDuration="3.016503747s" totalImagesPullingTime="976.485826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:32:00.014534775 +0000 UTC m=+559.400217879" watchObservedRunningTime="2026-09-16 21:32:00.016503747 +0000 UTC m=+559.402186831"
Sep 16 21:32:08.296769 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:32:08.296736    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:32:08.297149 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:32:08.296790    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:32:08.309139 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:32:08.309112    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:32:09.054758 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:32:09.054728    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p4qwx"
Sep 16 21:32:12.682519 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:32:12.682475    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:32:22.875964 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:32:22.875926    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:32:31.493425 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:32:31.493363    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:33:56.855329 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:56.855023    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-03acef"]
Sep 16 21:33:56.860012 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:56.859984    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-03acef"
Sep 16 21:33:56.860962 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:56.860937    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-03acef"]
Sep 16 21:33:56.862199 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:56.862174    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8588c\""
Sep 16 21:33:57.017935 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:57.017898    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8gm\" (UniqueName: \"kubernetes.io/projected/471bfcee-2324-4c1b-8c94-c473647aaeea-kube-api-access-4g8gm\") pod \"test-curl-7384-03acef\" (UID: \"471bfcee-2324-4c1b-8c94-c473647aaeea\") " pod="opendatahub/test-curl-7384-03acef"
Sep 16 21:33:57.118588 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:57.118506    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8gm\" (UniqueName: \"kubernetes.io/projected/471bfcee-2324-4c1b-8c94-c473647aaeea-kube-api-access-4g8gm\") pod \"test-curl-7384-03acef\" (UID: \"471bfcee-2324-4c1b-8c94-c473647aaeea\") " pod="opendatahub/test-curl-7384-03acef"
Sep 16 21:33:57.126076 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:57.126052    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8gm\" (UniqueName: \"kubernetes.io/projected/471bfcee-2324-4c1b-8c94-c473647aaeea-kube-api-access-4g8gm\") pod \"test-curl-7384-03acef\" (UID: \"471bfcee-2324-4c1b-8c94-c473647aaeea\") " pod="opendatahub/test-curl-7384-03acef"
Sep 16 21:33:57.170074 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:57.170046    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-03acef"
Sep 16 21:33:57.439939 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:57.439914    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-03acef"]
Sep 16 21:33:57.442380 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:33:57.442344    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod471bfcee_2324_4c1b_8c94_c473647aaeea.slice/crio-cc67fc32e11a40e0634916e789a28a195a46e9c150550414780deb5170769b05 WatchSource:0}: Error finding container cc67fc32e11a40e0634916e789a28a195a46e9c150550414780deb5170769b05: Status 404 returned error can't find the container with id cc67fc32e11a40e0634916e789a28a195a46e9c150550414780deb5170769b05
Sep 16 21:33:57.444180 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:57.444164    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:33:57.660184 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:57.660147    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-03acef" event={"ID":"471bfcee-2324-4c1b-8c94-c473647aaeea","Type":"ContainerStarted","Data":"cc67fc32e11a40e0634916e789a28a195a46e9c150550414780deb5170769b05"}
Sep 16 21:33:59.045935 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:59.045896    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:33:59.684003 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:59.683910    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-03acef" event={"ID":"471bfcee-2324-4c1b-8c94-c473647aaeea","Type":"ContainerStarted","Data":"2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee"}
Sep 16 21:33:59.705674 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:33:59.705626    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7384-03acef" podStartSLOduration=1.817105016 podStartE2EDuration="3.705611208s" totalImagesPullingTime="1.888506192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:33:59.703909042 +0000 UTC m=+679.089592125" watchObservedRunningTime="2026-09-16 21:33:59.705611208 +0000 UTC m=+679.091294285"
Sep 16 21:34:00.416556 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.416475    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-03acef"]
Sep 16 21:34:00.429165 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.429128    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-03acef"]
Sep 16 21:34:00.541894 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.541861    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-46b369"]
Sep 16 21:34:00.542252 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.542240    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471bfcee-2324-4c1b-8c94-c473647aaeea" containerName="test-curl-7384-03acef"
Sep 16 21:34:00.542296 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.542254    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="471bfcee-2324-4c1b-8c94-c473647aaeea" containerName="test-curl-7384-03acef"
Sep 16 21:34:00.542330 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.542311    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="471bfcee-2324-4c1b-8c94-c473647aaeea" containerName="test-curl-7384-03acef"
Sep 16 21:34:00.545466 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.545447    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-46b369"
Sep 16 21:34:00.552413 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.552373    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-46b369"]
Sep 16 21:34:00.651362 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.651327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw89j\" (UniqueName: \"kubernetes.io/projected/f6e5a797-67f6-4255-b4f7-58610aef3912-kube-api-access-pw89j\") pod \"test-curl-7384-46b369\" (UID: \"f6e5a797-67f6-4255-b4f7-58610aef3912\") " pod="opendatahub/test-curl-7384-46b369"
Sep 16 21:34:00.690985 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:34:00.690900    2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7384-03acef" container="test-curl-7384-03acef"
Sep 16 21:34:00.752052 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.752012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw89j\" (UniqueName: \"kubernetes.io/projected/f6e5a797-67f6-4255-b4f7-58610aef3912-kube-api-access-pw89j\") pod \"test-curl-7384-46b369\" (UID: \"f6e5a797-67f6-4255-b4f7-58610aef3912\") " pod="opendatahub/test-curl-7384-46b369"
Sep 16 21:34:00.760304 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.760280    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw89j\" (UniqueName: \"kubernetes.io/projected/f6e5a797-67f6-4255-b4f7-58610aef3912-kube-api-access-pw89j\") pod \"test-curl-7384-46b369\" (UID: \"f6e5a797-67f6-4255-b4f7-58610aef3912\") " pod="opendatahub/test-curl-7384-46b369"
Sep 16 21:34:00.849833 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.849798    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"]
Sep 16 21:34:00.853371 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.853353    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:34:00.855500 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.855476    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e5568d\""
Sep 16 21:34:00.855623 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.855512    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e5568d-dockercfg-m7vr8\""
Sep 16 21:34:00.856000 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.855980    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-46b369"
Sep 16 21:34:00.866092 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.866069    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"]
Sep 16 21:34:00.935176 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.935123    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"]
Sep 16 21:34:00.940227 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.940203    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:34:00.945991 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.945759    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"]
Sep 16 21:34:00.953150 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.953128    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0a6186-19c9-4c74-8082-999432b88093-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8\" (UID: \"1d0a6186-19c9-4c74-8082-999432b88093\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:34:00.953278 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:00.953250    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7j66\" (UniqueName: \"kubernetes.io/projected/1d0a6186-19c9-4c74-8082-999432b88093-kube-api-access-r7j66\") pod \"payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8\" (UID: \"1d0a6186-19c9-4c74-8082-999432b88093\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:34:01.054158 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.054121    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0a6186-19c9-4c74-8082-999432b88093-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8\" (UID: \"1d0a6186-19c9-4c74-8082-999432b88093\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:34:01.054316 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.054202    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47vx\" (UniqueName: \"kubernetes.io/projected/80a4fe71-7147-4091-831e-adc9a450c576-kube-api-access-h47vx\") pod \"payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm\" (UID: \"80a4fe71-7147-4091-831e-adc9a450c576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:34:01.054316 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.054237    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a4fe71-7147-4091-831e-adc9a450c576-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm\" (UID: \"80a4fe71-7147-4091-831e-adc9a450c576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:34:01.054316 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.054256    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7j66\" (UniqueName: \"kubernetes.io/projected/1d0a6186-19c9-4c74-8082-999432b88093-kube-api-access-r7j66\") pod \"payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8\" (UID: \"1d0a6186-19c9-4c74-8082-999432b88093\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:34:01.055026 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.054993    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0a6186-19c9-4c74-8082-999432b88093-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8\" (UID: \"1d0a6186-19c9-4c74-8082-999432b88093\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:34:01.072499 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.072472    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7j66\" (UniqueName: \"kubernetes.io/projected/1d0a6186-19c9-4c74-8082-999432b88093-kube-api-access-r7j66\") pod \"payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8\" (UID: \"1d0a6186-19c9-4c74-8082-999432b88093\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:34:01.134419 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.134376    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-46b369"]
Sep 16 21:34:01.136793 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:01.136754    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6e5a797_67f6_4255_b4f7_58610aef3912.slice/crio-225c183a89c46f36d6b94e11aff6c3bb67f0b9790b45127009b25063123aed05 WatchSource:0}: Error finding container 225c183a89c46f36d6b94e11aff6c3bb67f0b9790b45127009b25063123aed05: Status 404 returned error can't find the container with id 225c183a89c46f36d6b94e11aff6c3bb67f0b9790b45127009b25063123aed05
Sep 16 21:34:01.155715 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.155679    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47vx\" (UniqueName: \"kubernetes.io/projected/80a4fe71-7147-4091-831e-adc9a450c576-kube-api-access-h47vx\") pod \"payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm\" (UID: \"80a4fe71-7147-4091-831e-adc9a450c576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:34:01.155823 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.155737    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a4fe71-7147-4091-831e-adc9a450c576-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm\" (UID: \"80a4fe71-7147-4091-831e-adc9a450c576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:34:01.157070 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.157042    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a4fe71-7147-4091-831e-adc9a450c576-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm\" (UID: \"80a4fe71-7147-4091-831e-adc9a450c576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:34:01.164225 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.164151    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:34:01.172714 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.172641    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47vx\" (UniqueName: \"kubernetes.io/projected/80a4fe71-7147-4091-831e-adc9a450c576-kube-api-access-h47vx\") pod \"payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm\" (UID: \"80a4fe71-7147-4091-831e-adc9a450c576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:34:01.254145 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.254118    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:34:01.477094 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.477065    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"]
Sep 16 21:34:01.478939 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:01.478906    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0a6186_19c9_4c74_8082_999432b88093.slice/crio-f596f0798b2537d6c007f57fc8ddb8fcffe4214ed17754537b7b9f36492a5501 WatchSource:0}: Error finding container f596f0798b2537d6c007f57fc8ddb8fcffe4214ed17754537b7b9f36492a5501: Status 404 returned error can't find the container with id f596f0798b2537d6c007f57fc8ddb8fcffe4214ed17754537b7b9f36492a5501
Sep 16 21:34:01.548349 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.548321    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"]
Sep 16 21:34:01.549928 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:01.549903    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a4fe71_7147_4091_831e_adc9a450c576.slice/crio-a6d6b5add5ec8f9b1c0eeadd174cccb9a875820dda3f7287b3f257270d4de8f4 WatchSource:0}: Error finding container a6d6b5add5ec8f9b1c0eeadd174cccb9a875820dda3f7287b3f257270d4de8f4: Status 404 returned error can't find the container with id a6d6b5add5ec8f9b1c0eeadd174cccb9a875820dda3f7287b3f257270d4de8f4
Sep 16 21:34:01.699392 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.699354    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm" event={"ID":"80a4fe71-7147-4091-831e-adc9a450c576","Type":"ContainerStarted","Data":"66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446"}
Sep 16 21:34:01.699619 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.699436    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm" event={"ID":"80a4fe71-7147-4091-831e-adc9a450c576","Type":"ContainerStarted","Data":"a6d6b5add5ec8f9b1c0eeadd174cccb9a875820dda3f7287b3f257270d4de8f4"}
Sep 16 21:34:01.699619 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.699473    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:34:01.701023 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.700998    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8" event={"ID":"1d0a6186-19c9-4c74-8082-999432b88093","Type":"ContainerStarted","Data":"d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6"}
Sep 16 21:34:01.701129 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.701038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8" event={"ID":"1d0a6186-19c9-4c74-8082-999432b88093","Type":"ContainerStarted","Data":"f596f0798b2537d6c007f57fc8ddb8fcffe4214ed17754537b7b9f36492a5501"}
Sep 16 21:34:01.701129 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.701104    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:34:01.702759 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.702740    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-46b369" event={"ID":"f6e5a797-67f6-4255-b4f7-58610aef3912","Type":"ContainerStarted","Data":"c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824"}
Sep 16 21:34:01.702870 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.702768    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-46b369" event={"ID":"f6e5a797-67f6-4255-b4f7-58610aef3912","Type":"ContainerStarted","Data":"225c183a89c46f36d6b94e11aff6c3bb67f0b9790b45127009b25063123aed05"}
Sep 16 21:34:01.702870 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.702766    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7384-03acef" podUID="471bfcee-2324-4c1b-8c94-c473647aaeea" containerName="test-curl-7384-03acef" containerID="cri-o://2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee" gracePeriod=2
Sep 16 21:34:01.728922 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.728833    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm" podStartSLOduration=1.7288164190000002 podStartE2EDuration="1.728816419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:01.726532186 +0000 UTC m=+681.112215271" watchObservedRunningTime="2026-09-16 21:34:01.728816419 +0000 UTC m=+681.114499504"
Sep 16 21:34:01.764279 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.764238    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8" podStartSLOduration=1.764223743 podStartE2EDuration="1.764223743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:01.76268182 +0000 UTC m=+681.148364931" watchObservedRunningTime="2026-09-16 21:34:01.764223743 +0000 UTC m=+681.149906818"
Sep 16 21:34:01.788302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:01.788260    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7384-46b369" podStartSLOduration=1.7882503299999999 podStartE2EDuration="1.78825033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:01.787062233 +0000 UTC m=+681.172745309" watchObservedRunningTime="2026-09-16 21:34:01.78825033 +0000 UTC m=+681.173933474"
Sep 16 21:34:02.063672 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.063587    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"]
Sep 16 21:34:02.067316 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.067294    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:34:02.069527 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.069507    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-35f5fd-dockercfg-nlt9k\""
Sep 16 21:34:02.069614 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.069507    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-35f5fd\""
Sep 16 21:34:02.083134 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.083113    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"]
Sep 16 21:34:02.086997 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.086978    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"]
Sep 16 21:34:02.087091 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.087054    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:34:02.098924 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.098904    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"]
Sep 16 21:34:02.164682 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.164634    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8\" (UID: \"c78fcf47-01b0-4506-9115-5ec7eefaf5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:34:02.164682 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.164681    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttpbm\" (UniqueName: \"kubernetes.io/projected/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-kube-api-access-ttpbm\") pod \"payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8\" (UID: \"c78fcf47-01b0-4506-9115-5ec7eefaf5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:34:02.265576 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.265541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwmbc\" (UniqueName: \"kubernetes.io/projected/725c29ad-36fa-4062-b369-4e2ee78500d7-kube-api-access-fwmbc\") pod \"payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw\" (UID: \"725c29ad-36fa-4062-b369-4e2ee78500d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:34:02.265760 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.265621    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725c29ad-36fa-4062-b369-4e2ee78500d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw\" (UID: \"725c29ad-36fa-4062-b369-4e2ee78500d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:34:02.265896 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.265876    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8\" (UID: \"c78fcf47-01b0-4506-9115-5ec7eefaf5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:34:02.265967 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.265926    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttpbm\" (UniqueName: \"kubernetes.io/projected/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-kube-api-access-ttpbm\") pod \"payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8\" (UID: \"c78fcf47-01b0-4506-9115-5ec7eefaf5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:34:02.266809 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.266785    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8\" (UID: \"c78fcf47-01b0-4506-9115-5ec7eefaf5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:34:02.275932 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.275911    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttpbm\" (UniqueName: \"kubernetes.io/projected/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-kube-api-access-ttpbm\") pod \"payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8\" (UID: \"c78fcf47-01b0-4506-9115-5ec7eefaf5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:34:02.367213 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.367120    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwmbc\" (UniqueName: \"kubernetes.io/projected/725c29ad-36fa-4062-b369-4e2ee78500d7-kube-api-access-fwmbc\") pod \"payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw\" (UID: \"725c29ad-36fa-4062-b369-4e2ee78500d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:34:02.367213 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.367177    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725c29ad-36fa-4062-b369-4e2ee78500d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw\" (UID: \"725c29ad-36fa-4062-b369-4e2ee78500d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:34:02.367902 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.367878    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725c29ad-36fa-4062-b369-4e2ee78500d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw\" (UID: \"725c29ad-36fa-4062-b369-4e2ee78500d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:34:02.378181 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.378152    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:34:02.381150 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.381129    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwmbc\" (UniqueName: \"kubernetes.io/projected/725c29ad-36fa-4062-b369-4e2ee78500d7-kube-api-access-fwmbc\") pod \"payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw\" (UID: \"725c29ad-36fa-4062-b369-4e2ee78500d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:34:02.400962 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.400936    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:34:02.434886 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.431575    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-46b369"]
Sep 16 21:34:02.441372 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.441332    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-46b369"]
Sep 16 21:34:02.679794 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.679720    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"]
Sep 16 21:34:02.686151 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:02.686110    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78fcf47_01b0_4506_9115_5ec7eefaf5e1.slice/crio-c789dd94cf134cc490e402eaa60ed7f43c27c37c50402e2abf6e80ac80922884 WatchSource:0}: Error finding container c789dd94cf134cc490e402eaa60ed7f43c27c37c50402e2abf6e80ac80922884: Status 404 returned error can't find the container with id c789dd94cf134cc490e402eaa60ed7f43c27c37c50402e2abf6e80ac80922884
Sep 16 21:34:02.710948 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.710916    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8" event={"ID":"c78fcf47-01b0-4506-9115-5ec7eefaf5e1","Type":"ContainerStarted","Data":"c789dd94cf134cc490e402eaa60ed7f43c27c37c50402e2abf6e80ac80922884"}
Sep 16 21:34:02.711076 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:34:02.711059    2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7384-46b369" container="test-curl-7384-46b369"
Sep 16 21:34:02.729303 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:02.729275    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725c29ad_36fa_4062_b369_4e2ee78500d7.slice/crio-5518ad395f1db5261305aa99961ee96bc6d92b52b142921513529b80e6a6c37d WatchSource:0}: Error finding container 5518ad395f1db5261305aa99961ee96bc6d92b52b142921513529b80e6a6c37d: Status 404 returned error can't find the container with id 5518ad395f1db5261305aa99961ee96bc6d92b52b142921513529b80e6a6c37d
Sep 16 21:34:02.731577 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:02.731549    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"]
Sep 16 21:34:03.719213 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.719179    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8" event={"ID":"c78fcf47-01b0-4506-9115-5ec7eefaf5e1","Type":"ContainerStarted","Data":"5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8"}
Sep 16 21:34:03.719555 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.719285    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:34:03.720871 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.720847    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw" event={"ID":"725c29ad-36fa-4062-b369-4e2ee78500d7","Type":"ContainerStarted","Data":"fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37"}
Sep 16 21:34:03.720981 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.720887    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw" event={"ID":"725c29ad-36fa-4062-b369-4e2ee78500d7","Type":"ContainerStarted","Data":"5518ad395f1db5261305aa99961ee96bc6d92b52b142921513529b80e6a6c37d"}
Sep 16 21:34:03.720981 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.720941    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:34:03.721097 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.721006    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7384-46b369" podUID="f6e5a797-67f6-4255-b4f7-58610aef3912" containerName="test-curl-7384-46b369" containerID="cri-o://c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824" gracePeriod=2
Sep 16 21:34:03.744246 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.744197    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8" podStartSLOduration=1.744179666 podStartE2EDuration="1.744179666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:03.743110803 +0000 UTC m=+683.128793911" watchObservedRunningTime="2026-09-16 21:34:03.744179666 +0000 UTC m=+683.129862751"
Sep 16 21:34:03.763077 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.762983    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw" podStartSLOduration=1.7629654430000001 podStartE2EDuration="1.762965443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:03.762720762 +0000 UTC m=+683.148403839" watchObservedRunningTime="2026-09-16 21:34:03.762965443 +0000 UTC m=+683.148648530"
Sep 16 21:34:03.845917 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.845885    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"]
Sep 16 21:34:03.846299 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.846286    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e5a797-67f6-4255-b4f7-58610aef3912" containerName="test-curl-7384-46b369"
Sep 16 21:34:03.846366 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.846301    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e5a797-67f6-4255-b4f7-58610aef3912" containerName="test-curl-7384-46b369"
Sep 16 21:34:03.846428 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.846369    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e5a797-67f6-4255-b4f7-58610aef3912" containerName="test-curl-7384-46b369"
Sep 16 21:34:03.849652 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.849633    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:34:03.852486 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.852462    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-e047d8-dockercfg-d69dp\""
Sep 16 21:34:03.857518 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.857499    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-e047d8\""
Sep 16 21:34:03.867543 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.867519    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"]
Sep 16 21:34:03.883107 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.883086    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mg4f\" (UniqueName: \"kubernetes.io/projected/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-kube-api-access-5mg4f\") pod \"payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v\" (UID: \"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:34:03.883230 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.883207    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v\" (UID: \"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:34:03.895918 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.895901    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-03acef"
Sep 16 21:34:03.983998 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.983919    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g8gm\" (UniqueName: \"kubernetes.io/projected/471bfcee-2324-4c1b-8c94-c473647aaeea-kube-api-access-4g8gm\") pod \"471bfcee-2324-4c1b-8c94-c473647aaeea\" (UID: \"471bfcee-2324-4c1b-8c94-c473647aaeea\") "
Sep 16 21:34:03.984156 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.984138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mg4f\" (UniqueName: \"kubernetes.io/projected/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-kube-api-access-5mg4f\") pod \"payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v\" (UID: \"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:34:03.984207 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.984174    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v\" (UID: \"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:34:03.984978 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.984952    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v\" (UID: \"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:34:03.986104 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.986080    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471bfcee-2324-4c1b-8c94-c473647aaeea-kube-api-access-4g8gm" (OuterVolumeSpecName: "kube-api-access-4g8gm") pod "471bfcee-2324-4c1b-8c94-c473647aaeea" (UID: "471bfcee-2324-4c1b-8c94-c473647aaeea"). InnerVolumeSpecName "kube-api-access-4g8gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:34:03.994085 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:03.994060    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mg4f\" (UniqueName: \"kubernetes.io/projected/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-kube-api-access-5mg4f\") pod \"payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v\" (UID: \"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:34:04.085320 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.085283    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8gm\" (UniqueName: \"kubernetes.io/projected/471bfcee-2324-4c1b-8c94-c473647aaeea-kube-api-access-4g8gm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:34:04.162873 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.162837    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:34:04.458456 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.458368    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"]
Sep 16 21:34:04.460499 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:04.460466    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ecd51e_7c26_4d3f_83ae_ddb9c4f5dc6f.slice/crio-6f2101b647b09347ba612317355b6f66e4ce8cfae825112682561b53a5fba7ab WatchSource:0}: Error finding container 6f2101b647b09347ba612317355b6f66e4ce8cfae825112682561b53a5fba7ab: Status 404 returned error can't find the container with id 6f2101b647b09347ba612317355b6f66e4ce8cfae825112682561b53a5fba7ab
Sep 16 21:34:04.730187 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.730096    2567 generic.go:356] "Generic (PLEG): container finished" podID="471bfcee-2324-4c1b-8c94-c473647aaeea" containerID="2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee" exitCode=137
Sep 16 21:34:04.730187 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.730145    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-03acef"
Sep 16 21:34:04.730695 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.730195    2567 scope.go:117] "RemoveContainer" containerID="2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee"
Sep 16 21:34:04.732695 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.732579    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v" event={"ID":"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f","Type":"ContainerStarted","Data":"e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46"}
Sep 16 21:34:04.732695 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.732631    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v" event={"ID":"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f","Type":"ContainerStarted","Data":"6f2101b647b09347ba612317355b6f66e4ce8cfae825112682561b53a5fba7ab"}
Sep 16 21:34:04.733170 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.732978    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:34:04.741366 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.741342    2567 scope.go:117] "RemoveContainer" containerID="2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee"
Sep 16 21:34:04.741673 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:34:04.741650    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee\": container with ID starting with 2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee not found: ID does not exist" containerID="2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee"
Sep 16 21:34:04.741729 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.741683    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee"} err="failed to get container status \"2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee\": rpc error: code = NotFound desc = could not find container \"2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee\": container with ID starting with 2234e0b8c6a60c974d8aa1e3eab8b1e535501f563bfed01d16403911bac688ee not found: ID does not exist"
Sep 16 21:34:04.766083 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:34:04.766049    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7384-03acef\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="471bfcee-2324-4c1b-8c94-c473647aaeea" pod="opendatahub/test-curl-7384-03acef"
Sep 16 21:34:04.766389 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:04.766354    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v" podStartSLOduration=1.766340847 podStartE2EDuration="1.766340847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:04.764164114 +0000 UTC m=+684.149847256" watchObservedRunningTime="2026-09-16 21:34:04.766340847 +0000 UTC m=+684.152023930"
Sep 16 21:34:05.109857 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:05.109777    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="471bfcee-2324-4c1b-8c94-c473647aaeea" path="/var/lib/kubelet/pods/471bfcee-2324-4c1b-8c94-c473647aaeea/volumes"
Sep 16 21:34:05.878253 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:05.878231    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-46b369"
Sep 16 21:34:05.899864 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:05.899824    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw89j\" (UniqueName: \"kubernetes.io/projected/f6e5a797-67f6-4255-b4f7-58610aef3912-kube-api-access-pw89j\") pod \"f6e5a797-67f6-4255-b4f7-58610aef3912\" (UID: \"f6e5a797-67f6-4255-b4f7-58610aef3912\") "
Sep 16 21:34:05.902058 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:05.902028    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e5a797-67f6-4255-b4f7-58610aef3912-kube-api-access-pw89j" (OuterVolumeSpecName: "kube-api-access-pw89j") pod "f6e5a797-67f6-4255-b4f7-58610aef3912" (UID: "f6e5a797-67f6-4255-b4f7-58610aef3912"). InnerVolumeSpecName "kube-api-access-pw89j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:34:06.000754 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:06.000681    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw89j\" (UniqueName: \"kubernetes.io/projected/f6e5a797-67f6-4255-b4f7-58610aef3912-kube-api-access-pw89j\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:34:06.749542 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:06.749506    2567 generic.go:356] "Generic (PLEG): container finished" podID="f6e5a797-67f6-4255-b4f7-58610aef3912" containerID="c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824" exitCode=137
Sep 16 21:34:06.749707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:06.749559    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-46b369"
Sep 16 21:34:06.749707 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:06.749611    2567 scope.go:117] "RemoveContainer" containerID="c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824"
Sep 16 21:34:06.759391 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:06.759371    2567 scope.go:117] "RemoveContainer" containerID="c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824"
Sep 16 21:34:06.759688 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:34:06.759672    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824\": container with ID starting with c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824 not found: ID does not exist" containerID="c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824"
Sep 16 21:34:06.759859 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:06.759695    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824"} err="failed to get container status \"c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824\": rpc error: code = NotFound desc = could not find container \"c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824\": container with ID starting with c49a54be90b90fe033b092cf0f722030ee67525fab7d12a64f52ea53df2d6824 not found: ID does not exist"
Sep 16 21:34:06.761468 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:34:06.761449    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7384-46b369\" is forbidden: User \"system:node:ip-10-0-140-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-93.ec2.internal' and this object" podUID="f6e5a797-67f6-4255-b4f7-58610aef3912" pod="opendatahub/test-curl-7384-46b369"
Sep 16 21:34:07.114816 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:07.114735    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e5a797-67f6-4255-b4f7-58610aef3912" path="/var/lib/kubelet/pods/f6e5a797-67f6-4255-b4f7-58610aef3912/volumes"
Sep 16 21:34:09.449258 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.449203    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"]
Sep 16 21:34:09.454881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.454856    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:34:09.458193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.458156    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ddbefa\""
Sep 16 21:34:09.458193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.458176    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ddbefa-dockercfg-gmwnh\""
Sep 16 21:34:09.461371 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.461348    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"]
Sep 16 21:34:09.465229 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.465202    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:34:09.476015 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.475991    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"]
Sep 16 21:34:09.500462 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.500357    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"]
Sep 16 21:34:09.532668 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.532639    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77b62dba-a54d-4309-9b5b-ac961148f3dc-plugins-config-volume\") pod \"payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7\" (UID: \"77b62dba-a54d-4309-9b5b-ac961148f3dc\") " pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:34:09.532817 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.532703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b974z\" (UniqueName: \"kubernetes.io/projected/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-kube-api-access-b974z\") pod \"payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v\" (UID: \"bbeab1a8-ba6a-435e-97a1-47befb8c3c74\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:34:09.532817 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.532733    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v\" (UID: \"bbeab1a8-ba6a-435e-97a1-47befb8c3c74\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:34:09.532817 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.532755    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z882c\" (UniqueName: \"kubernetes.io/projected/77b62dba-a54d-4309-9b5b-ac961148f3dc-kube-api-access-z882c\") pod \"payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7\" (UID: \"77b62dba-a54d-4309-9b5b-ac961148f3dc\") " pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:34:09.634182 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.634145    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b974z\" (UniqueName: \"kubernetes.io/projected/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-kube-api-access-b974z\") pod \"payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v\" (UID: \"bbeab1a8-ba6a-435e-97a1-47befb8c3c74\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:34:09.634182 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.634186    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v\" (UID: \"bbeab1a8-ba6a-435e-97a1-47befb8c3c74\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:34:09.634558 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.634208    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z882c\" (UniqueName: \"kubernetes.io/projected/77b62dba-a54d-4309-9b5b-ac961148f3dc-kube-api-access-z882c\") pod \"payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7\" (UID: \"77b62dba-a54d-4309-9b5b-ac961148f3dc\") " pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:34:09.634558 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.634268    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77b62dba-a54d-4309-9b5b-ac961148f3dc-plugins-config-volume\") pod \"payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7\" (UID: \"77b62dba-a54d-4309-9b5b-ac961148f3dc\") " pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:34:09.635016 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.634996    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77b62dba-a54d-4309-9b5b-ac961148f3dc-plugins-config-volume\") pod \"payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7\" (UID: \"77b62dba-a54d-4309-9b5b-ac961148f3dc\") " pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:34:09.635132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.635106    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v\" (UID: \"bbeab1a8-ba6a-435e-97a1-47befb8c3c74\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:34:09.654534 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.654499    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z882c\" (UniqueName: \"kubernetes.io/projected/77b62dba-a54d-4309-9b5b-ac961148f3dc-kube-api-access-z882c\") pod \"payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7\" (UID: \"77b62dba-a54d-4309-9b5b-ac961148f3dc\") " pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:34:09.668165 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.668133    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b974z\" (UniqueName: \"kubernetes.io/projected/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-kube-api-access-b974z\") pod \"payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v\" (UID: \"bbeab1a8-ba6a-435e-97a1-47befb8c3c74\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:34:09.765355 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.765272    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:34:09.775392 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:09.775363    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:34:10.090075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:10.089353    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"]
Sep 16 21:34:10.091082 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:10.091050    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b62dba_a54d_4309_9b5b_ac961148f3dc.slice/crio-953d7a01cb780cc91d520e47b0731d4620ffc7fb69065538e532cefe8e925534 WatchSource:0}: Error finding container 953d7a01cb780cc91d520e47b0731d4620ffc7fb69065538e532cefe8e925534: Status 404 returned error can't find the container with id 953d7a01cb780cc91d520e47b0731d4620ffc7fb69065538e532cefe8e925534
Sep 16 21:34:10.094544 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:10.094519    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"]
Sep 16 21:34:10.094800 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:10.094780    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbeab1a8_ba6a_435e_97a1_47befb8c3c74.slice/crio-fe6856e379f14a91dff9d59e0630d73e4897cc0568bdc87e8650fa0e324abcc9 WatchSource:0}: Error finding container fe6856e379f14a91dff9d59e0630d73e4897cc0568bdc87e8650fa0e324abcc9: Status 404 returned error can't find the container with id fe6856e379f14a91dff9d59e0630d73e4897cc0568bdc87e8650fa0e324abcc9
Sep 16 21:34:10.776733 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:10.776697    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7" event={"ID":"77b62dba-a54d-4309-9b5b-ac961148f3dc","Type":"ContainerStarted","Data":"10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd"}
Sep 16 21:34:10.777151 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:10.776755    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:34:10.777151 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:10.776783    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7" event={"ID":"77b62dba-a54d-4309-9b5b-ac961148f3dc","Type":"ContainerStarted","Data":"953d7a01cb780cc91d520e47b0731d4620ffc7fb69065538e532cefe8e925534"}
Sep 16 21:34:10.778473 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:10.778443    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v" event={"ID":"bbeab1a8-ba6a-435e-97a1-47befb8c3c74","Type":"ContainerStarted","Data":"01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de"}
Sep 16 21:34:10.778594 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:10.778492    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v" event={"ID":"bbeab1a8-ba6a-435e-97a1-47befb8c3c74","Type":"ContainerStarted","Data":"fe6856e379f14a91dff9d59e0630d73e4897cc0568bdc87e8650fa0e324abcc9"}
Sep 16 21:34:10.778594 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:10.778563    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:34:10.836267 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:10.836220    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v" podStartSLOduration=1.836202646 podStartE2EDuration="1.836202646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:10.835166045 +0000 UTC m=+690.220849129" watchObservedRunningTime="2026-09-16 21:34:10.836202646 +0000 UTC m=+690.221885723"
Sep 16 21:34:10.837652 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:10.837607    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7" podStartSLOduration=1.837592127 podStartE2EDuration="1.837592127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:10.810554034 +0000 UTC m=+690.196237138" watchObservedRunningTime="2026-09-16 21:34:10.837592127 +0000 UTC m=+690.223275208"
Sep 16 21:34:12.713245 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:12.713205    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:34:12.713686 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:12.713296    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:34:14.734579 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:14.734547    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:34:14.735059 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:14.734855    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:34:15.415808 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:15.415766    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:34:16.751539 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:16.751508    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:34:17.332721 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.332683    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"]
Sep 16 21:34:17.336314 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.336293    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:34:17.347611 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.347591    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e078853\""
Sep 16 21:34:17.348763 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.348748    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e078853-dockercfg-ztt2n\""
Sep 16 21:34:17.380933 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.380905    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"]
Sep 16 21:34:17.408923 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.408891    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jr7c\" (UniqueName: \"kubernetes.io/projected/05b0affd-098a-40de-92b5-da4f746576db-kube-api-access-2jr7c\") pod \"payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98\" (UID: \"05b0affd-098a-40de-92b5-da4f746576db\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:34:17.409068 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.408995    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05b0affd-098a-40de-92b5-da4f746576db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98\" (UID: \"05b0affd-098a-40de-92b5-da4f746576db\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:34:17.510291 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.510251    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05b0affd-098a-40de-92b5-da4f746576db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98\" (UID: \"05b0affd-098a-40de-92b5-da4f746576db\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:34:17.510514 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.510331    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jr7c\" (UniqueName: \"kubernetes.io/projected/05b0affd-098a-40de-92b5-da4f746576db-kube-api-access-2jr7c\") pod \"payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98\" (UID: \"05b0affd-098a-40de-92b5-da4f746576db\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:34:17.511025 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.510998    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05b0affd-098a-40de-92b5-da4f746576db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98\" (UID: \"05b0affd-098a-40de-92b5-da4f746576db\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:34:17.529467 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.529443    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jr7c\" (UniqueName: \"kubernetes.io/projected/05b0affd-098a-40de-92b5-da4f746576db-kube-api-access-2jr7c\") pod \"payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98\" (UID: \"05b0affd-098a-40de-92b5-da4f746576db\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:34:17.612863 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.612782    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"]
Sep 16 21:34:17.616549 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.616525    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:34:17.646585 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.646546    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:34:17.646938 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.646913    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"]
Sep 16 21:34:17.711984 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.711954    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d392a5-47a2-435c-9aa4-c5b79e952881-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x\" (UID: \"a6d392a5-47a2-435c-9aa4-c5b79e952881\") " pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:34:17.712129 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.711994    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5qkm\" (UniqueName: \"kubernetes.io/projected/a6d392a5-47a2-435c-9aa4-c5b79e952881-kube-api-access-k5qkm\") pod \"payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x\" (UID: \"a6d392a5-47a2-435c-9aa4-c5b79e952881\") " pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:34:17.812656 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.812623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d392a5-47a2-435c-9aa4-c5b79e952881-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x\" (UID: \"a6d392a5-47a2-435c-9aa4-c5b79e952881\") " pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:34:17.812656 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.812659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5qkm\" (UniqueName: \"kubernetes.io/projected/a6d392a5-47a2-435c-9aa4-c5b79e952881-kube-api-access-k5qkm\") pod \"payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x\" (UID: \"a6d392a5-47a2-435c-9aa4-c5b79e952881\") " pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:34:17.813522 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.813495    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d392a5-47a2-435c-9aa4-c5b79e952881-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x\" (UID: \"a6d392a5-47a2-435c-9aa4-c5b79e952881\") " pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:34:17.829723 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.829699    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5qkm\" (UniqueName: \"kubernetes.io/projected/a6d392a5-47a2-435c-9aa4-c5b79e952881-kube-api-access-k5qkm\") pod \"payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x\" (UID: \"a6d392a5-47a2-435c-9aa4-c5b79e952881\") " pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:34:17.928617 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.928541    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:34:17.950379 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:17.950355    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"]
Sep 16 21:34:17.951860 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:17.951829    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b0affd_098a_40de_92b5_da4f746576db.slice/crio-4ac1f85743694ee00dfccdb652ae32e76263e39b3b595a88c12b4c803f98cc31 WatchSource:0}: Error finding container 4ac1f85743694ee00dfccdb652ae32e76263e39b3b595a88c12b4c803f98cc31: Status 404 returned error can't find the container with id 4ac1f85743694ee00dfccdb652ae32e76263e39b3b595a88c12b4c803f98cc31
Sep 16 21:34:18.240519 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:18.240491    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"]
Sep 16 21:34:18.242667 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:18.242634    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d392a5_47a2_435c_9aa4_c5b79e952881.slice/crio-f8ecd1a84dbba2fc1e5693e937287ba11d787d2b50da7fe0eab5cc99aab79c9e WatchSource:0}: Error finding container f8ecd1a84dbba2fc1e5693e937287ba11d787d2b50da7fe0eab5cc99aab79c9e: Status 404 returned error can't find the container with id f8ecd1a84dbba2fc1e5693e937287ba11d787d2b50da7fe0eab5cc99aab79c9e
Sep 16 21:34:18.831769 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:18.831730    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98" event={"ID":"05b0affd-098a-40de-92b5-da4f746576db","Type":"ContainerStarted","Data":"d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89"}
Sep 16 21:34:18.832261 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:18.831799    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98" event={"ID":"05b0affd-098a-40de-92b5-da4f746576db","Type":"ContainerStarted","Data":"4ac1f85743694ee00dfccdb652ae32e76263e39b3b595a88c12b4c803f98cc31"}
Sep 16 21:34:18.832261 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:18.831933    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:34:18.833619 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:18.833595    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x" event={"ID":"a6d392a5-47a2-435c-9aa4-c5b79e952881","Type":"ContainerStarted","Data":"2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b"}
Sep 16 21:34:18.833717 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:18.833639    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x" event={"ID":"a6d392a5-47a2-435c-9aa4-c5b79e952881","Type":"ContainerStarted","Data":"f8ecd1a84dbba2fc1e5693e937287ba11d787d2b50da7fe0eab5cc99aab79c9e"}
Sep 16 21:34:18.833717 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:18.833685    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:34:18.980888 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:18.980834    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98" podStartSLOduration=1.9808175810000002 podStartE2EDuration="1.980817581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:18.912435251 +0000 UTC m=+698.298118356" watchObservedRunningTime="2026-09-16 21:34:18.980817581 +0000 UTC m=+698.366500716"
Sep 16 21:34:18.981503 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:18.981459    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x" podStartSLOduration=1.981447596 podStartE2EDuration="1.981447596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:18.977152064 +0000 UTC m=+698.362835149" watchObservedRunningTime="2026-09-16 21:34:18.981447596 +0000 UTC m=+698.367130679"
Sep 16 21:34:21.786700 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:21.786669    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:34:21.787062 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:21.786738    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:34:25.729088 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.729045    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"]
Sep 16 21:34:25.732949 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.732927    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:34:25.735611 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.735582    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-44e517\""
Sep 16 21:34:25.735937 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.735920    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-44e517-dockercfg-86m5k\""
Sep 16 21:34:25.742947 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.742926    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"]
Sep 16 21:34:25.799923 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.799884    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"]
Sep 16 21:34:25.803546 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.803524    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:34:25.821042 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.821015    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"]
Sep 16 21:34:25.888796 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.888767    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122898b2-56eb-4c82-9036-ddf9f6a246e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j\" (UID: \"122898b2-56eb-4c82-9036-ddf9f6a246e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:34:25.888967 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.888807    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e25bec0-1e2a-486e-832f-c76ead1393ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44e517-6f764db755-sshjh\" (UID: \"7e25bec0-1e2a-486e-832f-c76ead1393ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:34:25.888967 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.888825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjbdz\" (UniqueName: \"kubernetes.io/projected/7e25bec0-1e2a-486e-832f-c76ead1393ae-kube-api-access-cjbdz\") pod \"payload-processing-e2e-inf-b-44e517-6f764db755-sshjh\" (UID: \"7e25bec0-1e2a-486e-832f-c76ead1393ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:34:25.888967 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.888843    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r224l\" (UniqueName: \"kubernetes.io/projected/122898b2-56eb-4c82-9036-ddf9f6a246e4-kube-api-access-r224l\") pod \"payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j\" (UID: \"122898b2-56eb-4c82-9036-ddf9f6a246e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:34:25.989573 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.989485    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e25bec0-1e2a-486e-832f-c76ead1393ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44e517-6f764db755-sshjh\" (UID: \"7e25bec0-1e2a-486e-832f-c76ead1393ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:34:25.989573 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.989525    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjbdz\" (UniqueName: \"kubernetes.io/projected/7e25bec0-1e2a-486e-832f-c76ead1393ae-kube-api-access-cjbdz\") pod \"payload-processing-e2e-inf-b-44e517-6f764db755-sshjh\" (UID: \"7e25bec0-1e2a-486e-832f-c76ead1393ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:34:25.989573 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.989552    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r224l\" (UniqueName: \"kubernetes.io/projected/122898b2-56eb-4c82-9036-ddf9f6a246e4-kube-api-access-r224l\") pod \"payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j\" (UID: \"122898b2-56eb-4c82-9036-ddf9f6a246e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:34:25.989850 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.989693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122898b2-56eb-4c82-9036-ddf9f6a246e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j\" (UID: \"122898b2-56eb-4c82-9036-ddf9f6a246e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:34:25.990263 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.990235    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e25bec0-1e2a-486e-832f-c76ead1393ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44e517-6f764db755-sshjh\" (UID: \"7e25bec0-1e2a-486e-832f-c76ead1393ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:34:25.990431 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.990385    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122898b2-56eb-4c82-9036-ddf9f6a246e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j\" (UID: \"122898b2-56eb-4c82-9036-ddf9f6a246e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:34:25.998835 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:25.998812    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjbdz\" (UniqueName: \"kubernetes.io/projected/7e25bec0-1e2a-486e-832f-c76ead1393ae-kube-api-access-cjbdz\") pod \"payload-processing-e2e-inf-b-44e517-6f764db755-sshjh\" (UID: \"7e25bec0-1e2a-486e-832f-c76ead1393ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:34:26.001506 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.001483    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r224l\" (UniqueName: \"kubernetes.io/projected/122898b2-56eb-4c82-9036-ddf9f6a246e4-kube-api-access-r224l\") pod \"payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j\" (UID: \"122898b2-56eb-4c82-9036-ddf9f6a246e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:34:26.045988 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.045951    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:34:26.114147 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.114100    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:34:26.340829 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.340801    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"]
Sep 16 21:34:26.342326 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:26.342296    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122898b2_56eb_4c82_9036_ddf9f6a246e4.slice/crio-c53f14ccbc1d3f2391b4610c999f5c133104f57eb642b7c25a698e0f89e772cb WatchSource:0}: Error finding container c53f14ccbc1d3f2391b4610c999f5c133104f57eb642b7c25a698e0f89e772cb: Status 404 returned error can't find the container with id c53f14ccbc1d3f2391b4610c999f5c133104f57eb642b7c25a698e0f89e772cb
Sep 16 21:34:26.417045 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.417015    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"]
Sep 16 21:34:26.418918 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:34:26.418892    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e25bec0_1e2a_486e_832f_c76ead1393ae.slice/crio-234bccc85ff75d5a99dcb6010e8b8cdc81a4dcd44383e0127fca30e0f969a64e WatchSource:0}: Error finding container 234bccc85ff75d5a99dcb6010e8b8cdc81a4dcd44383e0127fca30e0f969a64e: Status 404 returned error can't find the container with id 234bccc85ff75d5a99dcb6010e8b8cdc81a4dcd44383e0127fca30e0f969a64e
Sep 16 21:34:26.891118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.891078    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh" event={"ID":"7e25bec0-1e2a-486e-832f-c76ead1393ae","Type":"ContainerStarted","Data":"e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a"}
Sep 16 21:34:26.891613 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.891136    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh" event={"ID":"7e25bec0-1e2a-486e-832f-c76ead1393ae","Type":"ContainerStarted","Data":"234bccc85ff75d5a99dcb6010e8b8cdc81a4dcd44383e0127fca30e0f969a64e"}
Sep 16 21:34:26.891613 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.891298    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:34:26.892824 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.892797    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j" event={"ID":"122898b2-56eb-4c82-9036-ddf9f6a246e4","Type":"ContainerStarted","Data":"2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290"}
Sep 16 21:34:26.892914 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.892842    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j" event={"ID":"122898b2-56eb-4c82-9036-ddf9f6a246e4","Type":"ContainerStarted","Data":"c53f14ccbc1d3f2391b4610c999f5c133104f57eb642b7c25a698e0f89e772cb"}
Sep 16 21:34:26.892914 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.892869    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:34:26.952352 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.952285    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh" podStartSLOduration=1.95222768 podStartE2EDuration="1.95222768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:26.932820004 +0000 UTC m=+706.318503112" watchObservedRunningTime="2026-09-16 21:34:26.95222768 +0000 UTC m=+706.337910766"
Sep 16 21:34:26.987687 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:26.987638    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j" podStartSLOduration=1.98762314 podStartE2EDuration="1.98762314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:26.985531275 +0000 UTC m=+706.371214362" watchObservedRunningTime="2026-09-16 21:34:26.98762314 +0000 UTC m=+706.373306225"
Sep 16 21:34:29.841832 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:29.841799    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:34:29.842274 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:29.841891    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:34:32.395389 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:32.395344    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:34:37.901640 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:37.901564    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:34:37.901640 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:34:37.901640    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:35:00.410080 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.410045    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"]
Sep 16 21:35:00.410604 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.410308    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x" podUID="a6d392a5-47a2-435c-9aa4-c5b79e952881" containerName="payload-processing" containerID="cri-o://2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b" gracePeriod=30
Sep 16 21:35:00.416766 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.416661    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"]
Sep 16 21:35:00.416958 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.416870    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98" podUID="05b0affd-098a-40de-92b5-da4f746576db" containerName="payload-pre-processing" containerID="cri-o://d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89" gracePeriod=30
Sep 16 21:35:00.700415 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.700380    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:35:00.703599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.703525    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:35:00.797975 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.797943    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jr7c\" (UniqueName: \"kubernetes.io/projected/05b0affd-098a-40de-92b5-da4f746576db-kube-api-access-2jr7c\") pod \"05b0affd-098a-40de-92b5-da4f746576db\" (UID: \"05b0affd-098a-40de-92b5-da4f746576db\") "
Sep 16 21:35:00.798132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.797991    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05b0affd-098a-40de-92b5-da4f746576db-plugins-config-volume\") pod \"05b0affd-098a-40de-92b5-da4f746576db\" (UID: \"05b0affd-098a-40de-92b5-da4f746576db\") "
Sep 16 21:35:00.798132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.798119    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d392a5-47a2-435c-9aa4-c5b79e952881-plugins-config-volume\") pod \"a6d392a5-47a2-435c-9aa4-c5b79e952881\" (UID: \"a6d392a5-47a2-435c-9aa4-c5b79e952881\") "
Sep 16 21:35:00.798222 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.798160    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5qkm\" (UniqueName: \"kubernetes.io/projected/a6d392a5-47a2-435c-9aa4-c5b79e952881-kube-api-access-k5qkm\") pod \"a6d392a5-47a2-435c-9aa4-c5b79e952881\" (UID: \"a6d392a5-47a2-435c-9aa4-c5b79e952881\") "
Sep 16 21:35:00.798326 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.798307    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b0affd-098a-40de-92b5-da4f746576db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05b0affd-098a-40de-92b5-da4f746576db" (UID: "05b0affd-098a-40de-92b5-da4f746576db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:35:00.798513 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.798492    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d392a5-47a2-435c-9aa4-c5b79e952881-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6d392a5-47a2-435c-9aa4-c5b79e952881" (UID: "a6d392a5-47a2-435c-9aa4-c5b79e952881"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:35:00.798604 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.798592    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05b0affd-098a-40de-92b5-da4f746576db-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:35:00.798656 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.798607    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d392a5-47a2-435c-9aa4-c5b79e952881-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:35:00.800000 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.799979    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b0affd-098a-40de-92b5-da4f746576db-kube-api-access-2jr7c" (OuterVolumeSpecName: "kube-api-access-2jr7c") pod "05b0affd-098a-40de-92b5-da4f746576db" (UID: "05b0affd-098a-40de-92b5-da4f746576db"). InnerVolumeSpecName "kube-api-access-2jr7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:35:00.800195 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.800175    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d392a5-47a2-435c-9aa4-c5b79e952881-kube-api-access-k5qkm" (OuterVolumeSpecName: "kube-api-access-k5qkm") pod "a6d392a5-47a2-435c-9aa4-c5b79e952881" (UID: "a6d392a5-47a2-435c-9aa4-c5b79e952881"). InnerVolumeSpecName "kube-api-access-k5qkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:35:00.899604 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.899571    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5qkm\" (UniqueName: \"kubernetes.io/projected/a6d392a5-47a2-435c-9aa4-c5b79e952881-kube-api-access-k5qkm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:35:00.899604 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:00.899599    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jr7c\" (UniqueName: \"kubernetes.io/projected/05b0affd-098a-40de-92b5-da4f746576db-kube-api-access-2jr7c\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:35:01.100920 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.100883    2567 generic.go:356] "Generic (PLEG): container finished" podID="05b0affd-098a-40de-92b5-da4f746576db" containerID="d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89" exitCode=0
Sep 16 21:35:01.101102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.100921    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98" event={"ID":"05b0affd-098a-40de-92b5-da4f746576db","Type":"ContainerDied","Data":"d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89"}
Sep 16 21:35:01.101102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.100974    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98" event={"ID":"05b0affd-098a-40de-92b5-da4f746576db","Type":"ContainerDied","Data":"4ac1f85743694ee00dfccdb652ae32e76263e39b3b595a88c12b4c803f98cc31"}
Sep 16 21:35:01.101102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.100929    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"
Sep 16 21:35:01.101102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.100997    2567 scope.go:117] "RemoveContainer" containerID="d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89"
Sep 16 21:35:01.102649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.102627    2567 generic.go:356] "Generic (PLEG): container finished" podID="a6d392a5-47a2-435c-9aa4-c5b79e952881" containerID="2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b" exitCode=0
Sep 16 21:35:01.102815 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.102799    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"
Sep 16 21:35:01.111274 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.111247    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x" event={"ID":"a6d392a5-47a2-435c-9aa4-c5b79e952881","Type":"ContainerDied","Data":"2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b"}
Sep 16 21:35:01.111415 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.111289    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x" event={"ID":"a6d392a5-47a2-435c-9aa4-c5b79e952881","Type":"ContainerDied","Data":"f8ecd1a84dbba2fc1e5693e937287ba11d787d2b50da7fe0eab5cc99aab79c9e"}
Sep 16 21:35:01.112524 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.112502    2567 scope.go:117] "RemoveContainer" containerID="d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89"
Sep 16 21:35:01.112812 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:35:01.112793    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89\": container with ID starting with d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89 not found: ID does not exist" containerID="d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89"
Sep 16 21:35:01.112872 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.112820    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89"} err="failed to get container status \"d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89\": rpc error: code = NotFound desc = could not find container \"d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89\": container with ID starting with d214c1eb576c8adea38a01f52bf361b6c7cc6462956d265b742185620f455c89 not found: ID does not exist"
Sep 16 21:35:01.112872 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.112838    2567 scope.go:117] "RemoveContainer" containerID="2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b"
Sep 16 21:35:01.121854 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.121836    2567 scope.go:117] "RemoveContainer" containerID="2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b"
Sep 16 21:35:01.122148 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:35:01.122120    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b\": container with ID starting with 2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b not found: ID does not exist" containerID="2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b"
Sep 16 21:35:01.122249 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.122157    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b"} err="failed to get container status \"2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b\": rpc error: code = NotFound desc = could not find container \"2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b\": container with ID starting with 2210f84353579c60853d3489fc4a5b03975af3a73c04b4d500bb7553e467c22b not found: ID does not exist"
Sep 16 21:35:01.133971 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.133941    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"]
Sep 16 21:35:01.136723 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.136691    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e078853-9f65f94c5-f9g6x"]
Sep 16 21:35:01.147987 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.147963    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"]
Sep 16 21:35:01.158309 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:01.158281    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e078853-7498845cf4-kbz98"]
Sep 16 21:35:03.108703 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:03.108668    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05b0affd-098a-40de-92b5-da4f746576db" path="/var/lib/kubelet/pods/05b0affd-098a-40de-92b5-da4f746576db/volumes"
Sep 16 21:35:03.109065 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:03.109023    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6d392a5-47a2-435c-9aa4-c5b79e952881" path="/var/lib/kubelet/pods/a6d392a5-47a2-435c-9aa4-c5b79e952881/volumes"
Sep 16 21:35:06.685570 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.685530    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:35:06.874251 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.874219    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"]
Sep 16 21:35:06.874755 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.874737    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05b0affd-098a-40de-92b5-da4f746576db" containerName="payload-pre-processing"
Sep 16 21:35:06.874863 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.874757    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="05b0affd-098a-40de-92b5-da4f746576db" containerName="payload-pre-processing"
Sep 16 21:35:06.874863 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.874794    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6d392a5-47a2-435c-9aa4-c5b79e952881" containerName="payload-processing"
Sep 16 21:35:06.874863 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.874803    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6d392a5-47a2-435c-9aa4-c5b79e952881" containerName="payload-processing"
Sep 16 21:35:06.875019 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.874917    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6d392a5-47a2-435c-9aa4-c5b79e952881" containerName="payload-processing"
Sep 16 21:35:06.875019 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.874934    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="05b0affd-098a-40de-92b5-da4f746576db" containerName="payload-pre-processing"
Sep 16 21:35:06.877891 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.877867    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:06.884078 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.884057    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-595d23dd\""
Sep 16 21:35:06.894227 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.894204    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"]
Sep 16 21:35:06.913461 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.913432    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"]
Sep 16 21:35:06.916974 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.916957    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:06.920221 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.920201    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-595d23dd-dockercfg-lfhpb\""
Sep 16 21:35:06.943646 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.943567    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"]
Sep 16 21:35:06.952958 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.952929    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225df059-e48e-44e4-9cc0-9f6c75e35b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd\" (UID: \"225df059-e48e-44e4-9cc0-9f6c75e35b97\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:06.953108 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:06.952969    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88s9w\" (UniqueName: \"kubernetes.io/projected/225df059-e48e-44e4-9cc0-9f6c75e35b97-kube-api-access-88s9w\") pod \"payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd\" (UID: \"225df059-e48e-44e4-9cc0-9f6c75e35b97\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:07.053866 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.053817    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225df059-e48e-44e4-9cc0-9f6c75e35b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd\" (UID: \"225df059-e48e-44e4-9cc0-9f6c75e35b97\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:07.053866 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.053877    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88s9w\" (UniqueName: \"kubernetes.io/projected/225df059-e48e-44e4-9cc0-9f6c75e35b97-kube-api-access-88s9w\") pod \"payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd\" (UID: \"225df059-e48e-44e4-9cc0-9f6c75e35b97\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:07.054075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.053932    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-plugins-config-volume\") pod \"payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj\" (UID: \"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747\") " pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:07.054075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.053952    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psnzt\" (UniqueName: \"kubernetes.io/projected/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-kube-api-access-psnzt\") pod \"payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj\" (UID: \"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747\") " pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:07.054663 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.054643    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225df059-e48e-44e4-9cc0-9f6c75e35b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd\" (UID: \"225df059-e48e-44e4-9cc0-9f6c75e35b97\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:07.063282 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.063260    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88s9w\" (UniqueName: \"kubernetes.io/projected/225df059-e48e-44e4-9cc0-9f6c75e35b97-kube-api-access-88s9w\") pod \"payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd\" (UID: \"225df059-e48e-44e4-9cc0-9f6c75e35b97\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:07.155624 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.155460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-plugins-config-volume\") pod \"payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj\" (UID: \"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747\") " pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:07.155624 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.155511    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psnzt\" (UniqueName: \"kubernetes.io/projected/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-kube-api-access-psnzt\") pod \"payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj\" (UID: \"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747\") " pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:07.156101 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.156080    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-plugins-config-volume\") pod \"payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj\" (UID: \"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747\") " pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:07.166613 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.166586    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psnzt\" (UniqueName: \"kubernetes.io/projected/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-kube-api-access-psnzt\") pod \"payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj\" (UID: \"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747\") " pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:07.188380 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.188359    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:07.229357 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.229327    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:07.465776 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.465746    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"]
Sep 16 21:35:07.467260 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:35:07.467226    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225df059_e48e_44e4_9cc0_9f6c75e35b97.slice/crio-c7e2c00509533370c122566fcfa83d7f2db745cac8290ad0739ac0cb20198598 WatchSource:0}: Error finding container c7e2c00509533370c122566fcfa83d7f2db745cac8290ad0739ac0cb20198598: Status 404 returned error can't find the container with id c7e2c00509533370c122566fcfa83d7f2db745cac8290ad0739ac0cb20198598
Sep 16 21:35:07.518788 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:35:07.518762    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7d6b70_20a8_4da3_9a8a_35fbac6f4747.slice/crio-ec5549ea15ef2ee11585402b7a6e93d9e7d7d755add42ee216e37df1d065e7a4 WatchSource:0}: Error finding container ec5549ea15ef2ee11585402b7a6e93d9e7d7d755add42ee216e37df1d065e7a4: Status 404 returned error can't find the container with id ec5549ea15ef2ee11585402b7a6e93d9e7d7d755add42ee216e37df1d065e7a4
Sep 16 21:35:07.518909 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:07.518886    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"]
Sep 16 21:35:08.156501 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:08.156467    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj" event={"ID":"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747","Type":"ContainerStarted","Data":"4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426"}
Sep 16 21:35:08.156968 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:08.156526    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj" event={"ID":"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747","Type":"ContainerStarted","Data":"ec5549ea15ef2ee11585402b7a6e93d9e7d7d755add42ee216e37df1d065e7a4"}
Sep 16 21:35:08.156968 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:08.156642    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:08.158343 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:08.158311    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd" event={"ID":"225df059-e48e-44e4-9cc0-9f6c75e35b97","Type":"ContainerStarted","Data":"b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7"}
Sep 16 21:35:08.158501 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:08.158361    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd" event={"ID":"225df059-e48e-44e4-9cc0-9f6c75e35b97","Type":"ContainerStarted","Data":"c7e2c00509533370c122566fcfa83d7f2db745cac8290ad0739ac0cb20198598"}
Sep 16 21:35:08.158501 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:08.158415    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:08.182531 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:08.182480    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj" podStartSLOduration=2.182465811 podStartE2EDuration="2.182465811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:35:08.179574072 +0000 UTC m=+747.565257181" watchObservedRunningTime="2026-09-16 21:35:08.182465811 +0000 UTC m=+747.568148931"
Sep 16 21:35:08.198112 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:08.198070    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd" podStartSLOduration=2.198054761 podStartE2EDuration="2.198054761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:35:08.196750834 +0000 UTC m=+747.582433941" watchObservedRunningTime="2026-09-16 21:35:08.198054761 +0000 UTC m=+747.583737891"
Sep 16 21:35:19.166660 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:19.166626    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:19.167033 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:19.166860    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:24.481177 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:24.481140    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:35:36.383735 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.383699    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"]
Sep 16 21:35:36.384329 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.383932    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj" podUID="1a7d6b70-20a8-4da3-9a8a-35fbac6f4747" containerName="payload-processing" containerID="cri-o://4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426" gracePeriod=30
Sep 16 21:35:36.391813 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.391525    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"]
Sep 16 21:35:36.391813 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.391739    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd" podUID="225df059-e48e-44e4-9cc0-9f6c75e35b97" containerName="payload-pre-processing" containerID="cri-o://b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7" gracePeriod=30
Sep 16 21:35:36.643256 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.643201    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:36.678955 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.678932    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:36.816005 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.815971    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88s9w\" (UniqueName: \"kubernetes.io/projected/225df059-e48e-44e4-9cc0-9f6c75e35b97-kube-api-access-88s9w\") pod \"225df059-e48e-44e4-9cc0-9f6c75e35b97\" (UID: \"225df059-e48e-44e4-9cc0-9f6c75e35b97\") "
Sep 16 21:35:36.816164 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.816021    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225df059-e48e-44e4-9cc0-9f6c75e35b97-plugins-config-volume\") pod \"225df059-e48e-44e4-9cc0-9f6c75e35b97\" (UID: \"225df059-e48e-44e4-9cc0-9f6c75e35b97\") "
Sep 16 21:35:36.816164 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.816052    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-plugins-config-volume\") pod \"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747\" (UID: \"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747\") "
Sep 16 21:35:36.816164 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.816127    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psnzt\" (UniqueName: \"kubernetes.io/projected/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-kube-api-access-psnzt\") pod \"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747\" (UID: \"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747\") "
Sep 16 21:35:36.816520 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.816486    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a7d6b70-20a8-4da3-9a8a-35fbac6f4747" (UID: "1a7d6b70-20a8-4da3-9a8a-35fbac6f4747"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:35:36.816643 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.816537    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225df059-e48e-44e4-9cc0-9f6c75e35b97-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "225df059-e48e-44e4-9cc0-9f6c75e35b97" (UID: "225df059-e48e-44e4-9cc0-9f6c75e35b97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:35:36.818688 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.818661    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-kube-api-access-psnzt" (OuterVolumeSpecName: "kube-api-access-psnzt") pod "1a7d6b70-20a8-4da3-9a8a-35fbac6f4747" (UID: "1a7d6b70-20a8-4da3-9a8a-35fbac6f4747"). InnerVolumeSpecName "kube-api-access-psnzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:35:36.818778 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.818717    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225df059-e48e-44e4-9cc0-9f6c75e35b97-kube-api-access-88s9w" (OuterVolumeSpecName: "kube-api-access-88s9w") pod "225df059-e48e-44e4-9cc0-9f6c75e35b97" (UID: "225df059-e48e-44e4-9cc0-9f6c75e35b97"). InnerVolumeSpecName "kube-api-access-88s9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:35:36.917630 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.917564    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psnzt\" (UniqueName: \"kubernetes.io/projected/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-kube-api-access-psnzt\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:35:36.917630 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.917588    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88s9w\" (UniqueName: \"kubernetes.io/projected/225df059-e48e-44e4-9cc0-9f6c75e35b97-kube-api-access-88s9w\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:35:36.917630 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.917597    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225df059-e48e-44e4-9cc0-9f6c75e35b97-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:35:36.917630 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:36.917607    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:35:37.340630 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.340593    2567 generic.go:356] "Generic (PLEG): container finished" podID="225df059-e48e-44e4-9cc0-9f6c75e35b97" containerID="b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7" exitCode=0
Sep 16 21:35:37.340826 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.340640    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd" event={"ID":"225df059-e48e-44e4-9cc0-9f6c75e35b97","Type":"ContainerDied","Data":"b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7"}
Sep 16 21:35:37.340826 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.340652    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"
Sep 16 21:35:37.340826 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.340701    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd" event={"ID":"225df059-e48e-44e4-9cc0-9f6c75e35b97","Type":"ContainerDied","Data":"c7e2c00509533370c122566fcfa83d7f2db745cac8290ad0739ac0cb20198598"}
Sep 16 21:35:37.340826 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.340728    2567 scope.go:117] "RemoveContainer" containerID="b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7"
Sep 16 21:35:37.342070 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.342045    2567 generic.go:356] "Generic (PLEG): container finished" podID="1a7d6b70-20a8-4da3-9a8a-35fbac6f4747" containerID="4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426" exitCode=0
Sep 16 21:35:37.342193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.342081    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj" event={"ID":"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747","Type":"ContainerDied","Data":"4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426"}
Sep 16 21:35:37.342193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.342099    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"
Sep 16 21:35:37.342193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.342118    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj" event={"ID":"1a7d6b70-20a8-4da3-9a8a-35fbac6f4747","Type":"ContainerDied","Data":"ec5549ea15ef2ee11585402b7a6e93d9e7d7d755add42ee216e37df1d065e7a4"}
Sep 16 21:35:37.351529 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.351514    2567 scope.go:117] "RemoveContainer" containerID="b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7"
Sep 16 21:35:37.351790 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:35:37.351773    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7\": container with ID starting with b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7 not found: ID does not exist" containerID="b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7"
Sep 16 21:35:37.351863 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.351797    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7"} err="failed to get container status \"b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7\": rpc error: code = NotFound desc = could not find container \"b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7\": container with ID starting with b5c06ff74f6c358e25e99eabec54da148640ed87bd937b0b43111b1cb05f61c7 not found: ID does not exist"
Sep 16 21:35:37.351863 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.351812    2567 scope.go:117] "RemoveContainer" containerID="4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426"
Sep 16 21:35:37.360347 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.360325    2567 scope.go:117] "RemoveContainer" containerID="4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426"
Sep 16 21:35:37.360617 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:35:37.360600    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426\": container with ID starting with 4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426 not found: ID does not exist" containerID="4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426"
Sep 16 21:35:37.360699 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.360624    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426"} err="failed to get container status \"4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426\": rpc error: code = NotFound desc = could not find container \"4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426\": container with ID starting with 4740c8f4783199966d463598f7f30157f5bd0126b0e0bee331016b452ed8d426 not found: ID does not exist"
Sep 16 21:35:37.361203 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.361182    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"]
Sep 16 21:35:37.370253 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.370232    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-595d23dd-5bf9587466-fz6rd"]
Sep 16 21:35:37.382137 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.382117    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"]
Sep 16 21:35:37.398787 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:37.398764    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-595d23dd-f7d5d9f4b-jp6jj"]
Sep 16 21:35:39.109208 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:39.109172    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a7d6b70-20a8-4da3-9a8a-35fbac6f4747" path="/var/lib/kubelet/pods/1a7d6b70-20a8-4da3-9a8a-35fbac6f4747/volumes"
Sep 16 21:35:39.109609 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:39.109549    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="225df059-e48e-44e4-9cc0-9f6c75e35b97" path="/var/lib/kubelet/pods/225df059-e48e-44e4-9cc0-9f6c75e35b97/volumes"
Sep 16 21:35:42.279531 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.279502    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:35:42.838810 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.838778    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"]
Sep 16 21:35:42.839232 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.839220    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7d6b70-20a8-4da3-9a8a-35fbac6f4747" containerName="payload-processing"
Sep 16 21:35:42.839277 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.839234    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7d6b70-20a8-4da3-9a8a-35fbac6f4747" containerName="payload-processing"
Sep 16 21:35:42.839277 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.839257    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="225df059-e48e-44e4-9cc0-9f6c75e35b97" containerName="payload-pre-processing"
Sep 16 21:35:42.839277 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.839263    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="225df059-e48e-44e4-9cc0-9f6c75e35b97" containerName="payload-pre-processing"
Sep 16 21:35:42.839369 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.839327    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7d6b70-20a8-4da3-9a8a-35fbac6f4747" containerName="payload-processing"
Sep 16 21:35:42.839369 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.839341    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="225df059-e48e-44e4-9cc0-9f6c75e35b97" containerName="payload-pre-processing"
Sep 16 21:35:42.843938 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.843919    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:35:42.846087 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.846067    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8eafdc51-dockercfg-2ckw9\""
Sep 16 21:35:42.846199 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.846104    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8eafdc51\""
Sep 16 21:35:42.851595 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.851570    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"]
Sep 16 21:35:42.864659 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.864634    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9tvg\" (UniqueName: \"kubernetes.io/projected/366e6036-88cf-420c-8e07-76934c2f2c89-kube-api-access-g9tvg\") pod \"payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x\" (UID: \"366e6036-88cf-420c-8e07-76934c2f2c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:35:42.864770 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.864701    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366e6036-88cf-420c-8e07-76934c2f2c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x\" (UID: \"366e6036-88cf-420c-8e07-76934c2f2c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:35:42.871980 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.871962    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"]
Sep 16 21:35:42.875458 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.875443    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:35:42.888492 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.888468    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"]
Sep 16 21:35:42.965151 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.965119    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366e6036-88cf-420c-8e07-76934c2f2c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x\" (UID: \"366e6036-88cf-420c-8e07-76934c2f2c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:35:42.965319 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.965173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-plugins-config-volume\") pod \"payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m\" (UID: \"d1436bcb-3489-4a0f-9f28-d26e40fcb83e\") " pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:35:42.965319 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.965236    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9tvg\" (UniqueName: \"kubernetes.io/projected/366e6036-88cf-420c-8e07-76934c2f2c89-kube-api-access-g9tvg\") pod \"payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x\" (UID: \"366e6036-88cf-420c-8e07-76934c2f2c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:35:42.965319 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.965256    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsnf\" (UniqueName: \"kubernetes.io/projected/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-kube-api-access-zmsnf\") pod \"payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m\" (UID: \"d1436bcb-3489-4a0f-9f28-d26e40fcb83e\") " pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:35:42.965907 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.965886    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366e6036-88cf-420c-8e07-76934c2f2c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x\" (UID: \"366e6036-88cf-420c-8e07-76934c2f2c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:35:42.972778 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:42.972756    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9tvg\" (UniqueName: \"kubernetes.io/projected/366e6036-88cf-420c-8e07-76934c2f2c89-kube-api-access-g9tvg\") pod \"payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x\" (UID: \"366e6036-88cf-420c-8e07-76934c2f2c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:35:43.066209 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:43.066168    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-plugins-config-volume\") pod \"payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m\" (UID: \"d1436bcb-3489-4a0f-9f28-d26e40fcb83e\") " pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:35:43.066374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:43.066259    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsnf\" (UniqueName: \"kubernetes.io/projected/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-kube-api-access-zmsnf\") pod \"payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m\" (UID: \"d1436bcb-3489-4a0f-9f28-d26e40fcb83e\") " pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:35:43.066874 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:43.066854    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-plugins-config-volume\") pod \"payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m\" (UID: \"d1436bcb-3489-4a0f-9f28-d26e40fcb83e\") " pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:35:43.073911 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:43.073888    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsnf\" (UniqueName: \"kubernetes.io/projected/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-kube-api-access-zmsnf\") pod \"payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m\" (UID: \"d1436bcb-3489-4a0f-9f28-d26e40fcb83e\") " pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:35:43.158324 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:43.158253    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:35:43.184338 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:43.184313    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:35:43.445494 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:43.445467    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"]
Sep 16 21:35:43.446490 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:35:43.446454    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366e6036_88cf_420c_8e07_76934c2f2c89.slice/crio-bb54c25104d3260b0924435fb3abff67e02e5b57ff5f35f101f1e244d1876c31 WatchSource:0}: Error finding container bb54c25104d3260b0924435fb3abff67e02e5b57ff5f35f101f1e244d1876c31: Status 404 returned error can't find the container with id bb54c25104d3260b0924435fb3abff67e02e5b57ff5f35f101f1e244d1876c31
Sep 16 21:35:43.489799 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:43.489771    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"]
Sep 16 21:35:43.490732 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:35:43.490706    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1436bcb_3489_4a0f_9f28_d26e40fcb83e.slice/crio-8ecefd3b51a28b32ca4177ac89ee9dc8eb7cea30aa59e41453042b13e4ce4573 WatchSource:0}: Error finding container 8ecefd3b51a28b32ca4177ac89ee9dc8eb7cea30aa59e41453042b13e4ce4573: Status 404 returned error can't find the container with id 8ecefd3b51a28b32ca4177ac89ee9dc8eb7cea30aa59e41453042b13e4ce4573
Sep 16 21:35:44.392757 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:44.392716    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x" event={"ID":"366e6036-88cf-420c-8e07-76934c2f2c89","Type":"ContainerStarted","Data":"54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3"}
Sep 16 21:35:44.392932 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:44.392783    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:35:44.392932 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:44.392809    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x" event={"ID":"366e6036-88cf-420c-8e07-76934c2f2c89","Type":"ContainerStarted","Data":"bb54c25104d3260b0924435fb3abff67e02e5b57ff5f35f101f1e244d1876c31"}
Sep 16 21:35:44.394227 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:44.394199    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m" event={"ID":"d1436bcb-3489-4a0f-9f28-d26e40fcb83e","Type":"ContainerStarted","Data":"384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24"}
Sep 16 21:35:44.394325 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:44.394249    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m" event={"ID":"d1436bcb-3489-4a0f-9f28-d26e40fcb83e","Type":"ContainerStarted","Data":"8ecefd3b51a28b32ca4177ac89ee9dc8eb7cea30aa59e41453042b13e4ce4573"}
Sep 16 21:35:44.394325 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:44.394278    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:35:44.411919 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:44.411880    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x" podStartSLOduration=2.411865699 podStartE2EDuration="2.411865699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:35:44.409360504 +0000 UTC m=+783.795043606" watchObservedRunningTime="2026-09-16 21:35:44.411865699 +0000 UTC m=+783.797548830"
Sep 16 21:35:44.430565 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:44.430505    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m" podStartSLOduration=2.430485218 podStartE2EDuration="2.430485218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:35:44.428935323 +0000 UTC m=+783.814618432" watchObservedRunningTime="2026-09-16 21:35:44.430485218 +0000 UTC m=+783.816168306"
Sep 16 21:35:47.269091 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.269052    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"]
Sep 16 21:35:47.272846 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.272829    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.275509 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.275483    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ddbefa\"/\"autores-model-ddbefa-kserve-self-signed-certs\""
Sep 16 21:35:47.275692 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.275644    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ddbefa\"/\"kube-root-ca.crt\""
Sep 16 21:35:47.275692 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.275644    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ddbefa\"/\"openshift-service-ca.crt\""
Sep 16 21:35:47.275849 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.275663    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ddbefa\"/\"default-dockercfg-7qfch\""
Sep 16 21:35:47.279836 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.279815    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"]
Sep 16 21:35:47.304476 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.304452    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tmp-dir\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.304600 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.304486    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-home\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.304600 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.304511    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-dshm\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.304713 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.304634    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkgg8\" (UniqueName: \"kubernetes.io/projected/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-kube-api-access-fkgg8\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.304713 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.304697    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-model-cache\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.304816 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.304760    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tls-certs\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.405530 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.405500    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tls-certs\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.405668 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.405539    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tmp-dir\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.405668 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.405563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-home\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.405750 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.405676    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-dshm\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.405811 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.405745    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkgg8\" (UniqueName: \"kubernetes.io/projected/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-kube-api-access-fkgg8\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.405811 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.405786    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-model-cache\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.405931 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.405912    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-home\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.406000 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.405960    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tmp-dir\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.406057 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.406036    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-model-cache\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.407922 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.407901    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-dshm\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.408296 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.408277    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tls-certs\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.413663 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.413643    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkgg8\" (UniqueName: \"kubernetes.io/projected/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-kube-api-access-fkgg8\") pod \"autores-model-ddbefa-kserve-7578fd859-87pjm\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") " pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.583901 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.583836    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:47.864299 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:47.864275    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"]
Sep 16 21:35:47.866272 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:35:47.866239    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2d5dcf9_fc8e_4d4b_b74f_f73b415b3223.slice/crio-aa553dedd74f4850b576cf8550e134213a1f8f7608c8d8d7f2435e4b323e2456 WatchSource:0}: Error finding container aa553dedd74f4850b576cf8550e134213a1f8f7608c8d8d7f2435e4b323e2456: Status 404 returned error can't find the container with id aa553dedd74f4850b576cf8550e134213a1f8f7608c8d8d7f2435e4b323e2456
Sep 16 21:35:48.422307 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:48.422269    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm" event={"ID":"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223","Type":"ContainerStarted","Data":"4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04"}
Sep 16 21:35:48.422756 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:48.422318    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm" event={"ID":"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223","Type":"ContainerStarted","Data":"aa553dedd74f4850b576cf8550e134213a1f8f7608c8d8d7f2435e4b323e2456"}
Sep 16 21:35:48.441390 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:48.441338    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm" podStartSLOduration=1.4413215560000001 podStartE2EDuration="1.441321556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:35:48.440327361 +0000 UTC m=+787.826010489" watchObservedRunningTime="2026-09-16 21:35:48.441321556 +0000 UTC m=+787.827004640"
Sep 16 21:35:55.403582 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:55.403549    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:35:55.404037 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:55.403781    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:35:57.584495 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:57.584460    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:57.584840 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:57.584520    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:57.597015 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:57.596989    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:35:58.491129 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:35:58.491103    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:36:07.287106 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:07.287016    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:36:12.034049 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.033018    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"]
Sep 16 21:36:12.034049 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.033381    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m" podUID="d1436bcb-3489-4a0f-9f28-d26e40fcb83e" containerName="payload-processing" containerID="cri-o://384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24" gracePeriod=30
Sep 16 21:36:12.039418 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.037378    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"]
Sep 16 21:36:12.039418 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.037660    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x" podUID="366e6036-88cf-420c-8e07-76934c2f2c89" containerName="payload-pre-processing" containerID="cri-o://54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3" gracePeriod=30
Sep 16 21:36:12.321943 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.321915    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:36:12.325144 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.325120    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:36:12.423372 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.423338    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9tvg\" (UniqueName: \"kubernetes.io/projected/366e6036-88cf-420c-8e07-76934c2f2c89-kube-api-access-g9tvg\") pod \"366e6036-88cf-420c-8e07-76934c2f2c89\" (UID: \"366e6036-88cf-420c-8e07-76934c2f2c89\") "
Sep 16 21:36:12.423565 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.423450    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-plugins-config-volume\") pod \"d1436bcb-3489-4a0f-9f28-d26e40fcb83e\" (UID: \"d1436bcb-3489-4a0f-9f28-d26e40fcb83e\") "
Sep 16 21:36:12.423565 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.423516    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmsnf\" (UniqueName: \"kubernetes.io/projected/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-kube-api-access-zmsnf\") pod \"d1436bcb-3489-4a0f-9f28-d26e40fcb83e\" (UID: \"d1436bcb-3489-4a0f-9f28-d26e40fcb83e\") "
Sep 16 21:36:12.423693 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.423620    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366e6036-88cf-420c-8e07-76934c2f2c89-plugins-config-volume\") pod \"366e6036-88cf-420c-8e07-76934c2f2c89\" (UID: \"366e6036-88cf-420c-8e07-76934c2f2c89\") "
Sep 16 21:36:12.423865 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.423836    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1436bcb-3489-4a0f-9f28-d26e40fcb83e" (UID: "d1436bcb-3489-4a0f-9f28-d26e40fcb83e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:12.423973 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.423945    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:12.424209 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.424171    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366e6036-88cf-420c-8e07-76934c2f2c89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "366e6036-88cf-420c-8e07-76934c2f2c89" (UID: "366e6036-88cf-420c-8e07-76934c2f2c89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:12.425954 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.425919    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366e6036-88cf-420c-8e07-76934c2f2c89-kube-api-access-g9tvg" (OuterVolumeSpecName: "kube-api-access-g9tvg") pod "366e6036-88cf-420c-8e07-76934c2f2c89" (UID: "366e6036-88cf-420c-8e07-76934c2f2c89"). InnerVolumeSpecName "kube-api-access-g9tvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:12.426072 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.426022    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-kube-api-access-zmsnf" (OuterVolumeSpecName: "kube-api-access-zmsnf") pod "d1436bcb-3489-4a0f-9f28-d26e40fcb83e" (UID: "d1436bcb-3489-4a0f-9f28-d26e40fcb83e"). InnerVolumeSpecName "kube-api-access-zmsnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:12.524556 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.524528    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366e6036-88cf-420c-8e07-76934c2f2c89-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:12.524556 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.524553    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9tvg\" (UniqueName: \"kubernetes.io/projected/366e6036-88cf-420c-8e07-76934c2f2c89-kube-api-access-g9tvg\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:12.524556 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.524562    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmsnf\" (UniqueName: \"kubernetes.io/projected/d1436bcb-3489-4a0f-9f28-d26e40fcb83e-kube-api-access-zmsnf\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:12.572462 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.572371    2567 generic.go:356] "Generic (PLEG): container finished" podID="d1436bcb-3489-4a0f-9f28-d26e40fcb83e" containerID="384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24" exitCode=0
Sep 16 21:36:12.572462 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.572426    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"
Sep 16 21:36:12.572462 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.572436    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m" event={"ID":"d1436bcb-3489-4a0f-9f28-d26e40fcb83e","Type":"ContainerDied","Data":"384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24"}
Sep 16 21:36:12.572685 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.572505    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m" event={"ID":"d1436bcb-3489-4a0f-9f28-d26e40fcb83e","Type":"ContainerDied","Data":"8ecefd3b51a28b32ca4177ac89ee9dc8eb7cea30aa59e41453042b13e4ce4573"}
Sep 16 21:36:12.572685 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.572541    2567 scope.go:117] "RemoveContainer" containerID="384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24"
Sep 16 21:36:12.573868 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.573845    2567 generic.go:356] "Generic (PLEG): container finished" podID="366e6036-88cf-420c-8e07-76934c2f2c89" containerID="54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3" exitCode=0
Sep 16 21:36:12.573951 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.573885    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"
Sep 16 21:36:12.573951 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.573926    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x" event={"ID":"366e6036-88cf-420c-8e07-76934c2f2c89","Type":"ContainerDied","Data":"54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3"}
Sep 16 21:36:12.574035 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.573968    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x" event={"ID":"366e6036-88cf-420c-8e07-76934c2f2c89","Type":"ContainerDied","Data":"bb54c25104d3260b0924435fb3abff67e02e5b57ff5f35f101f1e244d1876c31"}
Sep 16 21:36:12.582762 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.582744    2567 scope.go:117] "RemoveContainer" containerID="384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24"
Sep 16 21:36:12.583108 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:36:12.583071    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24\": container with ID starting with 384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24 not found: ID does not exist" containerID="384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24"
Sep 16 21:36:12.583178 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.583114    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24"} err="failed to get container status \"384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24\": rpc error: code = NotFound desc = could not find container \"384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24\": container with ID starting with 384f152ad6e7e30eb43338592cba97281a7978f8c8595ffd66690a50510cfe24 not found: ID does not exist"
Sep 16 21:36:12.583178 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.583138    2567 scope.go:117] "RemoveContainer" containerID="54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3"
Sep 16 21:36:12.591337 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.591322    2567 scope.go:117] "RemoveContainer" containerID="54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3"
Sep 16 21:36:12.591662 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:36:12.591647    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3\": container with ID starting with 54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3 not found: ID does not exist" containerID="54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3"
Sep 16 21:36:12.591714 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.591667    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3"} err="failed to get container status \"54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3\": rpc error: code = NotFound desc = could not find container \"54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3\": container with ID starting with 54d50bfdff411c675a975ab7b2d6d64a5ec8c6b120adc2e28984247e4b22a3a3 not found: ID does not exist"
Sep 16 21:36:12.603614 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.603593    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"]
Sep 16 21:36:12.613281 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.613263    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8eafdc51-7664789686-zbv8x"]
Sep 16 21:36:12.626060 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.626041    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"]
Sep 16 21:36:12.636515 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:12.636490    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8eafdc51-6f48d848f7-jmr6m"]
Sep 16 21:36:13.109186 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:13.109155    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="366e6036-88cf-420c-8e07-76934c2f2c89" path="/var/lib/kubelet/pods/366e6036-88cf-420c-8e07-76934c2f2c89/volumes"
Sep 16 21:36:13.109627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:13.109609    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1436bcb-3489-4a0f-9f28-d26e40fcb83e" path="/var/lib/kubelet/pods/d1436bcb-3489-4a0f-9f28-d26e40fcb83e/volumes"
Sep 16 21:36:18.848044 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.848011    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"]
Sep 16 21:36:18.848466 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.848453    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="366e6036-88cf-420c-8e07-76934c2f2c89" containerName="payload-pre-processing"
Sep 16 21:36:18.848466 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.848467    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="366e6036-88cf-420c-8e07-76934c2f2c89" containerName="payload-pre-processing"
Sep 16 21:36:18.848561 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.848476    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1436bcb-3489-4a0f-9f28-d26e40fcb83e" containerName="payload-processing"
Sep 16 21:36:18.848561 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.848482    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1436bcb-3489-4a0f-9f28-d26e40fcb83e" containerName="payload-processing"
Sep 16 21:36:18.848624 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.848564    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1436bcb-3489-4a0f-9f28-d26e40fcb83e" containerName="payload-processing"
Sep 16 21:36:18.848624 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.848576    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="366e6036-88cf-420c-8e07-76934c2f2c89" containerName="payload-pre-processing"
Sep 16 21:36:18.853599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.853579    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:18.857612 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.857588    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a1fb6d89\""
Sep 16 21:36:18.873800 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.873774    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"]
Sep 16 21:36:18.898248 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.898223    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"]
Sep 16 21:36:18.901856 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.901838    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:18.904371 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.904356    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a1fb6d89-dockercfg-g9ds9\""
Sep 16 21:36:18.913753 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.913731    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"]
Sep 16 21:36:18.983819 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.983781    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81065a73-1416-4f48-ae6a-9ca7ab954fb4-plugins-config-volume\") pod \"payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb\" (UID: \"81065a73-1416-4f48-ae6a-9ca7ab954fb4\") " pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:18.984022 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.983842    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4gh\" (UniqueName: \"kubernetes.io/projected/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-kube-api-access-8z4gh\") pod \"payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5\" (UID: \"a425ffc4-95d6-48f8-82c4-cc564eeae8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:18.984022 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.983889    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5\" (UID: \"a425ffc4-95d6-48f8-82c4-cc564eeae8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:18.984022 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:18.983921    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnp9z\" (UniqueName: \"kubernetes.io/projected/81065a73-1416-4f48-ae6a-9ca7ab954fb4-kube-api-access-qnp9z\") pod \"payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb\" (UID: \"81065a73-1416-4f48-ae6a-9ca7ab954fb4\") " pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:19.084782 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.084747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4gh\" (UniqueName: \"kubernetes.io/projected/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-kube-api-access-8z4gh\") pod \"payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5\" (UID: \"a425ffc4-95d6-48f8-82c4-cc564eeae8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:19.084782 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.084788    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5\" (UID: \"a425ffc4-95d6-48f8-82c4-cc564eeae8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:19.084983 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.084909    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnp9z\" (UniqueName: \"kubernetes.io/projected/81065a73-1416-4f48-ae6a-9ca7ab954fb4-kube-api-access-qnp9z\") pod \"payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb\" (UID: \"81065a73-1416-4f48-ae6a-9ca7ab954fb4\") " pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:19.085032 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.084986    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81065a73-1416-4f48-ae6a-9ca7ab954fb4-plugins-config-volume\") pod \"payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb\" (UID: \"81065a73-1416-4f48-ae6a-9ca7ab954fb4\") " pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:19.085544 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.085528    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5\" (UID: \"a425ffc4-95d6-48f8-82c4-cc564eeae8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:19.085633 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.085618    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81065a73-1416-4f48-ae6a-9ca7ab954fb4-plugins-config-volume\") pod \"payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb\" (UID: \"81065a73-1416-4f48-ae6a-9ca7ab954fb4\") " pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:19.097041 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.097016    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnp9z\" (UniqueName: \"kubernetes.io/projected/81065a73-1416-4f48-ae6a-9ca7ab954fb4-kube-api-access-qnp9z\") pod \"payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb\" (UID: \"81065a73-1416-4f48-ae6a-9ca7ab954fb4\") " pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:19.097132 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.097034    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4gh\" (UniqueName: \"kubernetes.io/projected/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-kube-api-access-8z4gh\") pod \"payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5\" (UID: \"a425ffc4-95d6-48f8-82c4-cc564eeae8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:19.165439 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.165322    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:19.212679 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.212645    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:19.454599 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.454573    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"]
Sep 16 21:36:19.456225 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:36:19.456201    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81065a73_1416_4f48_ae6a_9ca7ab954fb4.slice/crio-7c38fdff22cd25fc833194cabc8d4ec037530b061990c290c852585e779debaf WatchSource:0}: Error finding container 7c38fdff22cd25fc833194cabc8d4ec037530b061990c290c852585e779debaf: Status 404 returned error can't find the container with id 7c38fdff22cd25fc833194cabc8d4ec037530b061990c290c852585e779debaf
Sep 16 21:36:19.510486 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.510466    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"]
Sep 16 21:36:19.512967 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:36:19.512931    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda425ffc4_95d6_48f8_82c4_cc564eeae8e6.slice/crio-07b44215f8b150fb7c0d75c9c75dca4d173a381e03c9a9bf18323fdaee6996cd WatchSource:0}: Error finding container 07b44215f8b150fb7c0d75c9c75dca4d173a381e03c9a9bf18323fdaee6996cd: Status 404 returned error can't find the container with id 07b44215f8b150fb7c0d75c9c75dca4d173a381e03c9a9bf18323fdaee6996cd
Sep 16 21:36:19.628302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.628256    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5" event={"ID":"a425ffc4-95d6-48f8-82c4-cc564eeae8e6","Type":"ContainerStarted","Data":"6d741685492ccd35c83687cd554216d5062f38f1eb898348934f51538a853fe2"}
Sep 16 21:36:19.628549 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.628326    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:19.628549 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.628350    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5" event={"ID":"a425ffc4-95d6-48f8-82c4-cc564eeae8e6","Type":"ContainerStarted","Data":"07b44215f8b150fb7c0d75c9c75dca4d173a381e03c9a9bf18323fdaee6996cd"}
Sep 16 21:36:19.630173 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.630149    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb" event={"ID":"81065a73-1416-4f48-ae6a-9ca7ab954fb4","Type":"ContainerStarted","Data":"45b30247e6198e66d66880a250660f8567af8f1a256d7fb969db387bdd0000bc"}
Sep 16 21:36:19.630280 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.630201    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb" event={"ID":"81065a73-1416-4f48-ae6a-9ca7ab954fb4","Type":"ContainerStarted","Data":"7c38fdff22cd25fc833194cabc8d4ec037530b061990c290c852585e779debaf"}
Sep 16 21:36:19.630280 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.630236    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:19.645819 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.645774    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5" podStartSLOduration=1.645756242 podStartE2EDuration="1.645756242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:36:19.643125438 +0000 UTC m=+819.028808558" watchObservedRunningTime="2026-09-16 21:36:19.645756242 +0000 UTC m=+819.031439325"
Sep 16 21:36:19.659858 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:19.659815    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb" podStartSLOduration=1.6598022810000002 podStartE2EDuration="1.659802281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:36:19.657865563 +0000 UTC m=+819.043548676" watchObservedRunningTime="2026-09-16 21:36:19.659802281 +0000 UTC m=+819.045485365"
Sep 16 21:36:21.862484 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:21.862449    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"]
Sep 16 21:36:30.637860 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:30.637815    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:30.638250 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:30.637904    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:30.948875 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:30.948849    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"]
Sep 16 21:36:30.949125 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:30.949057    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7" podUID="77b62dba-a54d-4309-9b5b-ac961148f3dc" containerName="payload-processing" containerID="cri-o://10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd" gracePeriod=30
Sep 16 21:36:30.982174 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:30.982143    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"]
Sep 16 21:36:30.984433 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:30.982521    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v" podUID="bbeab1a8-ba6a-435e-97a1-47befb8c3c74" containerName="payload-pre-processing" containerID="cri-o://01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de" gracePeriod=30
Sep 16 21:36:31.287805 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.287776    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:36:31.291171 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.291148    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:36:31.291920 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.291901    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-plugins-config-volume\") pod \"bbeab1a8-ba6a-435e-97a1-47befb8c3c74\" (UID: \"bbeab1a8-ba6a-435e-97a1-47befb8c3c74\") "
Sep 16 21:36:31.291989 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.291975    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77b62dba-a54d-4309-9b5b-ac961148f3dc-plugins-config-volume\") pod \"77b62dba-a54d-4309-9b5b-ac961148f3dc\" (UID: \"77b62dba-a54d-4309-9b5b-ac961148f3dc\") "
Sep 16 21:36:31.292080 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.292061    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z882c\" (UniqueName: \"kubernetes.io/projected/77b62dba-a54d-4309-9b5b-ac961148f3dc-kube-api-access-z882c\") pod \"77b62dba-a54d-4309-9b5b-ac961148f3dc\" (UID: \"77b62dba-a54d-4309-9b5b-ac961148f3dc\") "
Sep 16 21:36:31.292197 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.292098    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b974z\" (UniqueName: \"kubernetes.io/projected/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-kube-api-access-b974z\") pod \"bbeab1a8-ba6a-435e-97a1-47befb8c3c74\" (UID: \"bbeab1a8-ba6a-435e-97a1-47befb8c3c74\") "
Sep 16 21:36:31.292340 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.292316    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbeab1a8-ba6a-435e-97a1-47befb8c3c74" (UID: "bbeab1a8-ba6a-435e-97a1-47befb8c3c74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:31.292431 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.292390    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b62dba-a54d-4309-9b5b-ac961148f3dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77b62dba-a54d-4309-9b5b-ac961148f3dc" (UID: "77b62dba-a54d-4309-9b5b-ac961148f3dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:31.294135 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.294103    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b62dba-a54d-4309-9b5b-ac961148f3dc-kube-api-access-z882c" (OuterVolumeSpecName: "kube-api-access-z882c") pod "77b62dba-a54d-4309-9b5b-ac961148f3dc" (UID: "77b62dba-a54d-4309-9b5b-ac961148f3dc"). InnerVolumeSpecName "kube-api-access-z882c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:31.294273 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.294252    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-kube-api-access-b974z" (OuterVolumeSpecName: "kube-api-access-b974z") pod "bbeab1a8-ba6a-435e-97a1-47befb8c3c74" (UID: "bbeab1a8-ba6a-435e-97a1-47befb8c3c74"). InnerVolumeSpecName "kube-api-access-b974z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:31.393448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.393419    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77b62dba-a54d-4309-9b5b-ac961148f3dc-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:31.393448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.393447    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z882c\" (UniqueName: \"kubernetes.io/projected/77b62dba-a54d-4309-9b5b-ac961148f3dc-kube-api-access-z882c\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:31.393629 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.393456    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b974z\" (UniqueName: \"kubernetes.io/projected/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-kube-api-access-b974z\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:31.393629 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.393465    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbeab1a8-ba6a-435e-97a1-47befb8c3c74-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:31.711157 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.711126    2567 generic.go:356] "Generic (PLEG): container finished" podID="77b62dba-a54d-4309-9b5b-ac961148f3dc" containerID="10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd" exitCode=0
Sep 16 21:36:31.711640 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.711165    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"
Sep 16 21:36:31.711640 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.711205    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7" event={"ID":"77b62dba-a54d-4309-9b5b-ac961148f3dc","Type":"ContainerDied","Data":"10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd"}
Sep 16 21:36:31.711640 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.711252    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7" event={"ID":"77b62dba-a54d-4309-9b5b-ac961148f3dc","Type":"ContainerDied","Data":"953d7a01cb780cc91d520e47b0731d4620ffc7fb69065538e532cefe8e925534"}
Sep 16 21:36:31.711640 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.711275    2567 scope.go:117] "RemoveContainer" containerID="10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd"
Sep 16 21:36:31.712716 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.712691    2567 generic.go:356] "Generic (PLEG): container finished" podID="bbeab1a8-ba6a-435e-97a1-47befb8c3c74" containerID="01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de" exitCode=0
Sep 16 21:36:31.712814 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.712728    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"
Sep 16 21:36:31.712814 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.712784    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v" event={"ID":"bbeab1a8-ba6a-435e-97a1-47befb8c3c74","Type":"ContainerDied","Data":"01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de"}
Sep 16 21:36:31.712919 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.712826    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v" event={"ID":"bbeab1a8-ba6a-435e-97a1-47befb8c3c74","Type":"ContainerDied","Data":"fe6856e379f14a91dff9d59e0630d73e4897cc0568bdc87e8650fa0e324abcc9"}
Sep 16 21:36:31.722353 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.722247    2567 scope.go:117] "RemoveContainer" containerID="10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd"
Sep 16 21:36:31.722616 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:36:31.722576    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd\": container with ID starting with 10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd not found: ID does not exist" containerID="10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd"
Sep 16 21:36:31.722717 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.722628    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd"} err="failed to get container status \"10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd\": rpc error: code = NotFound desc = could not find container \"10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd\": container with ID starting with 10b6a009a54caed009b10fcebb71f4247508ec33b0b1308b124bc6c80ba2c8dd not found: ID does not exist"
Sep 16 21:36:31.722717 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.722662    2567 scope.go:117] "RemoveContainer" containerID="01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de"
Sep 16 21:36:31.731472 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.731454    2567 scope.go:117] "RemoveContainer" containerID="01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de"
Sep 16 21:36:31.731730 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:36:31.731711    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de\": container with ID starting with 01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de not found: ID does not exist" containerID="01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de"
Sep 16 21:36:31.731795 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.731754    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de"} err="failed to get container status \"01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de\": rpc error: code = NotFound desc = could not find container \"01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de\": container with ID starting with 01d13bec1d1aa7672a7fd51e011d209bdba53c72f6be1f95d7dbf4f6e1e7e9de not found: ID does not exist"
Sep 16 21:36:31.740772 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.740751    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"]
Sep 16 21:36:31.743508 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.743490    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ddbefa-5747dc84fd-jvd6v"]
Sep 16 21:36:31.751848 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.751829    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"]
Sep 16 21:36:31.755892 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:31.755872    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ddbefa-8448f787b5-wgkv7"]
Sep 16 21:36:33.108777 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:33.108743    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77b62dba-a54d-4309-9b5b-ac961148f3dc" path="/var/lib/kubelet/pods/77b62dba-a54d-4309-9b5b-ac961148f3dc/volumes"
Sep 16 21:36:33.109153 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:33.109097    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbeab1a8-ba6a-435e-97a1-47befb8c3c74" path="/var/lib/kubelet/pods/bbeab1a8-ba6a-435e-97a1-47befb8c3c74/volumes"
Sep 16 21:36:36.088631 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.088593    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:36:36.773449 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.773408    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"]
Sep 16 21:36:36.773926 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.773907    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbeab1a8-ba6a-435e-97a1-47befb8c3c74" containerName="payload-pre-processing"
Sep 16 21:36:36.774018 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.773929    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbeab1a8-ba6a-435e-97a1-47befb8c3c74" containerName="payload-pre-processing"
Sep 16 21:36:36.774018 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.773953    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b62dba-a54d-4309-9b5b-ac961148f3dc" containerName="payload-processing"
Sep 16 21:36:36.774018 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.773961    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b62dba-a54d-4309-9b5b-ac961148f3dc" containerName="payload-processing"
Sep 16 21:36:36.774188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.774064    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbeab1a8-ba6a-435e-97a1-47befb8c3c74" containerName="payload-pre-processing"
Sep 16 21:36:36.774188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.774076    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="77b62dba-a54d-4309-9b5b-ac961148f3dc" containerName="payload-processing"
Sep 16 21:36:36.777329 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.777306    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"
Sep 16 21:36:36.779442 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.779424    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:36:36.779539 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.779427    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:36:36.783631 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.783608    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"]
Sep 16 21:36:36.838674 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.838646    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abceba8-796f-4835-80b7-a7b48277795e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82\" (UID: \"4abceba8-796f-4835-80b7-a7b48277795e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"
Sep 16 21:36:36.882277 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.882250    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm" podUID="a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" containerName="main" containerID="cri-o://4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04" gracePeriod=45
Sep 16 21:36:36.939730 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.939704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abceba8-796f-4835-80b7-a7b48277795e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82\" (UID: \"4abceba8-796f-4835-80b7-a7b48277795e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"
Sep 16 21:36:36.940328 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:36.940301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abceba8-796f-4835-80b7-a7b48277795e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82\" (UID: \"4abceba8-796f-4835-80b7-a7b48277795e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"
Sep 16 21:36:37.089709 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.089679    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"
Sep 16 21:36:37.147555 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.147524    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:36:37.242482 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.242456    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkgg8\" (UniqueName: \"kubernetes.io/projected/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-kube-api-access-fkgg8\") pod \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") "
Sep 16 21:36:37.242661 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.242496    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-home\") pod \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") "
Sep 16 21:36:37.242661 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.242535    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-model-cache\") pod \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") "
Sep 16 21:36:37.242661 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.242566    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-dshm\") pod \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") "
Sep 16 21:36:37.242661 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.242589    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tls-certs\") pod \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") "
Sep 16 21:36:37.242661 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.242639    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tmp-dir\") pod \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\" (UID: \"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223\") "
Sep 16 21:36:37.242956 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.242766    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-home" (OuterVolumeSpecName: "home") pod "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" (UID: "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:37.242956 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.242807    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" (UID: "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:37.243048 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.242962    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-model-cache\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:37.243048 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.242980    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-home\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:37.243326 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.243300    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" (UID: "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:37.244794 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.244772    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-kube-api-access-fkgg8" (OuterVolumeSpecName: "kube-api-access-fkgg8") pod "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" (UID: "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223"). InnerVolumeSpecName "kube-api-access-fkgg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:37.244879 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.244823    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" (UID: "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:36:37.244879 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.244829    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-dshm" (OuterVolumeSpecName: "dshm") pod "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" (UID: "a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:37.343980 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.343902    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tmp-dir\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:37.343980 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.343928    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkgg8\" (UniqueName: \"kubernetes.io/projected/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-kube-api-access-fkgg8\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:37.343980 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.343938    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-dshm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:37.343980 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.343948    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223-tls-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:37.365786 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.365760    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"]
Sep 16 21:36:37.366535 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:36:37.366510    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abceba8_796f_4835_80b7_a7b48277795e.slice/crio-a009c0bd4c6b2bfb4c3ed8de59fd432e77292679e25b3145c2c01733ffab61bb WatchSource:0}: Error finding container a009c0bd4c6b2bfb4c3ed8de59fd432e77292679e25b3145c2c01733ffab61bb: Status 404 returned error can't find the container with id a009c0bd4c6b2bfb4c3ed8de59fd432e77292679e25b3145c2c01733ffab61bb
Sep 16 21:36:37.757418 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.757346    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82" event={"ID":"4abceba8-796f-4835-80b7-a7b48277795e","Type":"ContainerStarted","Data":"a009c0bd4c6b2bfb4c3ed8de59fd432e77292679e25b3145c2c01733ffab61bb"}
Sep 16 21:36:37.758719 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.758691    2567 generic.go:356] "Generic (PLEG): container finished" podID="a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" containerID="4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04" exitCode=0
Sep 16 21:36:37.758871 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.758735    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"
Sep 16 21:36:37.758871 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.758757    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm" event={"ID":"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223","Type":"ContainerDied","Data":"4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04"}
Sep 16 21:36:37.758871 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.758795    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm" event={"ID":"a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223","Type":"ContainerDied","Data":"aa553dedd74f4850b576cf8550e134213a1f8f7608c8d8d7f2435e4b323e2456"}
Sep 16 21:36:37.758871 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.758816    2567 scope.go:117] "RemoveContainer" containerID="4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04"
Sep 16 21:36:37.768285 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.768269    2567 scope.go:117] "RemoveContainer" containerID="4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04"
Sep 16 21:36:37.768575 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:36:37.768557    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04\": container with ID starting with 4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04 not found: ID does not exist" containerID="4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04"
Sep 16 21:36:37.768627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.768584    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04"} err="failed to get container status \"4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04\": rpc error: code = NotFound desc = could not find container \"4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04\": container with ID starting with 4361285a53aad65298714abf1a64aa2e2ccbdfc2a3211d1fdeb243c87ee5ab04 not found: ID does not exist"
Sep 16 21:36:37.799387 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.799361    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"]
Sep 16 21:36:37.807480 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:37.807454    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-ddbefa/autores-model-ddbefa-kserve-7578fd859-87pjm"]
Sep 16 21:36:39.110816 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:39.110777    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" path="/var/lib/kubelet/pods/a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223/volumes"
Sep 16 21:36:40.782312 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:40.782277    2567 generic.go:356] "Generic (PLEG): container finished" podID="4abceba8-796f-4835-80b7-a7b48277795e" containerID="7e0a5719f1ec43bb162317f338c7469913b9ca89354b28fd55dd1a9010c1b6f5" exitCode=0
Sep 16 21:36:40.782740 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:40.782362    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82" event={"ID":"4abceba8-796f-4835-80b7-a7b48277795e","Type":"ContainerDied","Data":"7e0a5719f1ec43bb162317f338c7469913b9ca89354b28fd55dd1a9010c1b6f5"}
Sep 16 21:36:41.939311 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:41.939281    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"
Sep 16 21:36:41.986692 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:41.986659    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abceba8-796f-4835-80b7-a7b48277795e-maas-api-service-ca\") pod \"4abceba8-796f-4835-80b7-a7b48277795e\" (UID: \"4abceba8-796f-4835-80b7-a7b48277795e\") "
Sep 16 21:36:41.987058 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:41.987031    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abceba8-796f-4835-80b7-a7b48277795e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4abceba8-796f-4835-80b7-a7b48277795e" (UID: "4abceba8-796f-4835-80b7-a7b48277795e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:42.087981 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:42.087912    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abceba8-796f-4835-80b7-a7b48277795e-maas-api-service-ca\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:42.794810 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:42.794775    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82" event={"ID":"4abceba8-796f-4835-80b7-a7b48277795e","Type":"ContainerDied","Data":"a009c0bd4c6b2bfb4c3ed8de59fd432e77292679e25b3145c2c01733ffab61bb"}
Sep 16 21:36:42.794810 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:42.794799    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"
Sep 16 21:36:42.795025 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:42.794821    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a009c0bd4c6b2bfb4c3ed8de59fd432e77292679e25b3145c2c01733ffab61bb"
Sep 16 21:36:48.810443 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.810407    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"]
Sep 16 21:36:48.810851 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.810838    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4abceba8-796f-4835-80b7-a7b48277795e" containerName="revoke-keys"
Sep 16 21:36:48.810851 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.810852    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4abceba8-796f-4835-80b7-a7b48277795e" containerName="revoke-keys"
Sep 16 21:36:48.810927 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.810875    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" containerName="main"
Sep 16 21:36:48.810927 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.810880    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" containerName="main"
Sep 16 21:36:48.810996 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.810945    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4abceba8-796f-4835-80b7-a7b48277795e" containerName="revoke-keys"
Sep 16 21:36:48.810996 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.810954    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d5dcf9-fc8e-4d4b-b74f-f73b415b3223" containerName="main"
Sep 16 21:36:48.814451 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.814433    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:48.817274 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.817254    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"openshift-service-ca.crt\""
Sep 16 21:36:48.817743 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.817723    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"kube-root-ca.crt\""
Sep 16 21:36:48.818235 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.818220    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"default-dockercfg-hxdqm\""
Sep 16 21:36:48.818286 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.818245    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"premium-w5-e047d8-kserve-self-signed-certs\""
Sep 16 21:36:48.841651 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.841621    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"]
Sep 16 21:36:48.952898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.952861    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-model-cache\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:48.953096 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.952907    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b303241-58d2-4262-be41-db53acad4694-tls-certs\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:48.953096 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.952968    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-dshm\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:48.953096 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.953004    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh2vr\" (UniqueName: \"kubernetes.io/projected/8b303241-58d2-4262-be41-db53acad4694-kube-api-access-vh2vr\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:48.953096 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.953084    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-home\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:48.953274 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:48.953105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-tmp-dir\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.053921 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.053883    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-dshm\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.054082 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.053939    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh2vr\" (UniqueName: \"kubernetes.io/projected/8b303241-58d2-4262-be41-db53acad4694-kube-api-access-vh2vr\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.054082 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.054002    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-home\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.054198 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.054110    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-tmp-dir\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.054253 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.054218    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-model-cache\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.054253 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.054256    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b303241-58d2-4262-be41-db53acad4694-tls-certs\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.054424 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.054356    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-home\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.054617 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.054592    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-tmp-dir\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.054822 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.054651    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-model-cache\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.056701 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.056677    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-dshm\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.057075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.057053    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b303241-58d2-4262-be41-db53acad4694-tls-certs\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.061747 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.061694    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh2vr\" (UniqueName: \"kubernetes.io/projected/8b303241-58d2-4262-be41-db53acad4694-kube-api-access-vh2vr\") pod \"premium-w5-e047d8-kserve-56d6656c7f-x5bbz\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") " pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.125445 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.125418    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:49.426470 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.426442    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"]
Sep 16 21:36:49.841179 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.841143    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz" event={"ID":"8b303241-58d2-4262-be41-db53acad4694","Type":"ContainerStarted","Data":"dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb"}
Sep 16 21:36:49.841585 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.841185    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz" event={"ID":"8b303241-58d2-4262-be41-db53acad4694","Type":"ContainerStarted","Data":"2bcbbde3ea86071086e515cf0d0a64f0d100887a567a650df04307f3211a3ea8"}
Sep 16 21:36:49.861049 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:49.861005    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz" podStartSLOduration=1.860989883 podStartE2EDuration="1.860989883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:36:49.858575245 +0000 UTC m=+849.244258347" watchObservedRunningTime="2026-09-16 21:36:49.860989883 +0000 UTC m=+849.246673000"
Sep 16 21:36:51.334991 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.334954    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"]
Sep 16 21:36:51.338922 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.338898    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:36:51.341173 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.341149    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b91829-dockercfg-hm497\""
Sep 16 21:36:51.341281 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.341184    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b91829\""
Sep 16 21:36:51.351428 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.351387    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"]
Sep 16 21:36:51.356018 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.355994    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"]
Sep 16 21:36:51.359440 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.359421    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:36:51.369316 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.367715    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"]
Sep 16 21:36:51.479604 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.479568    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0409a320-4439-4df8-bfda-25e80bb71cf9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5\" (UID: \"0409a320-4439-4df8-bfda-25e80bb71cf9\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:36:51.479846 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.479673    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9948d9c9-87b8-4bc4-b696-0a36dc12578e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz\" (UID: \"9948d9c9-87b8-4bc4-b696-0a36dc12578e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:36:51.479846 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.479797    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t84cl\" (UniqueName: \"kubernetes.io/projected/9948d9c9-87b8-4bc4-b696-0a36dc12578e-kube-api-access-t84cl\") pod \"payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz\" (UID: \"9948d9c9-87b8-4bc4-b696-0a36dc12578e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:36:51.480038 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.479865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52grk\" (UniqueName: \"kubernetes.io/projected/0409a320-4439-4df8-bfda-25e80bb71cf9-kube-api-access-52grk\") pod \"payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5\" (UID: \"0409a320-4439-4df8-bfda-25e80bb71cf9\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:36:51.556923 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.556887    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"]
Sep 16 21:36:51.557150 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.557125    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb" podUID="81065a73-1416-4f48-ae6a-9ca7ab954fb4" containerName="payload-processing" containerID="cri-o://45b30247e6198e66d66880a250660f8567af8f1a256d7fb969db387bdd0000bc" gracePeriod=30
Sep 16 21:36:51.564365 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.564296    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"]
Sep 16 21:36:51.564582 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.564554    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5" podUID="a425ffc4-95d6-48f8-82c4-cc564eeae8e6" containerName="payload-pre-processing" containerID="cri-o://6d741685492ccd35c83687cd554216d5062f38f1eb898348934f51538a853fe2" gracePeriod=30
Sep 16 21:36:51.580789 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.580763    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9948d9c9-87b8-4bc4-b696-0a36dc12578e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz\" (UID: \"9948d9c9-87b8-4bc4-b696-0a36dc12578e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:36:51.580904 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.580863    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t84cl\" (UniqueName: \"kubernetes.io/projected/9948d9c9-87b8-4bc4-b696-0a36dc12578e-kube-api-access-t84cl\") pod \"payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz\" (UID: \"9948d9c9-87b8-4bc4-b696-0a36dc12578e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:36:51.580972 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.580937    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52grk\" (UniqueName: \"kubernetes.io/projected/0409a320-4439-4df8-bfda-25e80bb71cf9-kube-api-access-52grk\") pod \"payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5\" (UID: \"0409a320-4439-4df8-bfda-25e80bb71cf9\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:36:51.581046 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.580993    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0409a320-4439-4df8-bfda-25e80bb71cf9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5\" (UID: \"0409a320-4439-4df8-bfda-25e80bb71cf9\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:36:51.581609 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.581587    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9948d9c9-87b8-4bc4-b696-0a36dc12578e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz\" (UID: \"9948d9c9-87b8-4bc4-b696-0a36dc12578e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:36:51.581765 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.581745    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0409a320-4439-4df8-bfda-25e80bb71cf9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5\" (UID: \"0409a320-4439-4df8-bfda-25e80bb71cf9\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:36:51.588449 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.588380    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52grk\" (UniqueName: \"kubernetes.io/projected/0409a320-4439-4df8-bfda-25e80bb71cf9-kube-api-access-52grk\") pod \"payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5\" (UID: \"0409a320-4439-4df8-bfda-25e80bb71cf9\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:36:51.588449 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.588425    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t84cl\" (UniqueName: \"kubernetes.io/projected/9948d9c9-87b8-4bc4-b696-0a36dc12578e-kube-api-access-t84cl\") pod \"payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz\" (UID: \"9948d9c9-87b8-4bc4-b696-0a36dc12578e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:36:51.651817 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.651783    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:36:51.672810 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.672783    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:36:51.778197 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.778118    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"]
Sep 16 21:36:51.782243 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.782215    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a1fb6d89-dpk82"]
Sep 16 21:36:51.857351 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.857271    2567 generic.go:356] "Generic (PLEG): container finished" podID="a425ffc4-95d6-48f8-82c4-cc564eeae8e6" containerID="6d741685492ccd35c83687cd554216d5062f38f1eb898348934f51538a853fe2" exitCode=0
Sep 16 21:36:51.857351 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.857338    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5" event={"ID":"a425ffc4-95d6-48f8-82c4-cc564eeae8e6","Type":"ContainerDied","Data":"6d741685492ccd35c83687cd554216d5062f38f1eb898348934f51538a853fe2"}
Sep 16 21:36:51.859122 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.859096    2567 generic.go:356] "Generic (PLEG): container finished" podID="81065a73-1416-4f48-ae6a-9ca7ab954fb4" containerID="45b30247e6198e66d66880a250660f8567af8f1a256d7fb969db387bdd0000bc" exitCode=0
Sep 16 21:36:51.859250 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.859176    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb" event={"ID":"81065a73-1416-4f48-ae6a-9ca7ab954fb4","Type":"ContainerDied","Data":"45b30247e6198e66d66880a250660f8567af8f1a256d7fb969db387bdd0000bc"}
Sep 16 21:36:51.894191 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.894171    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:51.903661 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.903632    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:51.937195 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.937171    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"]
Sep 16 21:36:51.938779 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:36:51.938745    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9948d9c9_87b8_4bc4_b696_0a36dc12578e.slice/crio-b0465c0339daff5526ca18c66c2f907d7c5c8e4566522e219b92bd52d6bf5a25 WatchSource:0}: Error finding container b0465c0339daff5526ca18c66c2f907d7c5c8e4566522e219b92bd52d6bf5a25: Status 404 returned error can't find the container with id b0465c0339daff5526ca18c66c2f907d7c5c8e4566522e219b92bd52d6bf5a25
Sep 16 21:36:51.972335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.972308    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"]
Sep 16 21:36:51.975587 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:36:51.975554    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0409a320_4439_4df8_bfda_25e80bb71cf9.slice/crio-ca563aff2f625b1ef0782481aa0374d29abbc8d44338ec32c7d4e11239c148e3 WatchSource:0}: Error finding container ca563aff2f625b1ef0782481aa0374d29abbc8d44338ec32c7d4e11239c148e3: Status 404 returned error can't find the container with id ca563aff2f625b1ef0782481aa0374d29abbc8d44338ec32c7d4e11239c148e3
Sep 16 21:36:51.984572 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.984552    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81065a73-1416-4f48-ae6a-9ca7ab954fb4-plugins-config-volume\") pod \"81065a73-1416-4f48-ae6a-9ca7ab954fb4\" (UID: \"81065a73-1416-4f48-ae6a-9ca7ab954fb4\") "
Sep 16 21:36:51.984661 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.984580    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnp9z\" (UniqueName: \"kubernetes.io/projected/81065a73-1416-4f48-ae6a-9ca7ab954fb4-kube-api-access-qnp9z\") pod \"81065a73-1416-4f48-ae6a-9ca7ab954fb4\" (UID: \"81065a73-1416-4f48-ae6a-9ca7ab954fb4\") "
Sep 16 21:36:51.984661 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.984617    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z4gh\" (UniqueName: \"kubernetes.io/projected/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-kube-api-access-8z4gh\") pod \"a425ffc4-95d6-48f8-82c4-cc564eeae8e6\" (UID: \"a425ffc4-95d6-48f8-82c4-cc564eeae8e6\") "
Sep 16 21:36:51.984850 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.984825    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-plugins-config-volume\") pod \"a425ffc4-95d6-48f8-82c4-cc564eeae8e6\" (UID: \"a425ffc4-95d6-48f8-82c4-cc564eeae8e6\") "
Sep 16 21:36:51.985118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.985004    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81065a73-1416-4f48-ae6a-9ca7ab954fb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81065a73-1416-4f48-ae6a-9ca7ab954fb4" (UID: "81065a73-1416-4f48-ae6a-9ca7ab954fb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:51.985224 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.985197    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a425ffc4-95d6-48f8-82c4-cc564eeae8e6" (UID: "a425ffc4-95d6-48f8-82c4-cc564eeae8e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:51.986589 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.986562    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81065a73-1416-4f48-ae6a-9ca7ab954fb4-kube-api-access-qnp9z" (OuterVolumeSpecName: "kube-api-access-qnp9z") pod "81065a73-1416-4f48-ae6a-9ca7ab954fb4" (UID: "81065a73-1416-4f48-ae6a-9ca7ab954fb4"). InnerVolumeSpecName "kube-api-access-qnp9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:51.986667 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:51.986602    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-kube-api-access-8z4gh" (OuterVolumeSpecName: "kube-api-access-8z4gh") pod "a425ffc4-95d6-48f8-82c4-cc564eeae8e6" (UID: "a425ffc4-95d6-48f8-82c4-cc564eeae8e6"). InnerVolumeSpecName "kube-api-access-8z4gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:52.086180 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.086139    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:52.086180 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.086168    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81065a73-1416-4f48-ae6a-9ca7ab954fb4-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:52.086180 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.086178    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnp9z\" (UniqueName: \"kubernetes.io/projected/81065a73-1416-4f48-ae6a-9ca7ab954fb4-kube-api-access-qnp9z\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:52.086180 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.086188    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z4gh\" (UniqueName: \"kubernetes.io/projected/a425ffc4-95d6-48f8-82c4-cc564eeae8e6-kube-api-access-8z4gh\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:36:52.867089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.867006    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"
Sep 16 21:36:52.867089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.867010    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5" event={"ID":"a425ffc4-95d6-48f8-82c4-cc564eeae8e6","Type":"ContainerDied","Data":"07b44215f8b150fb7c0d75c9c75dca4d173a381e03c9a9bf18323fdaee6996cd"}
Sep 16 21:36:52.867089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.867080    2567 scope.go:117] "RemoveContainer" containerID="6d741685492ccd35c83687cd554216d5062f38f1eb898348934f51538a853fe2"
Sep 16 21:36:52.869649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.869159    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz" event={"ID":"9948d9c9-87b8-4bc4-b696-0a36dc12578e","Type":"ContainerStarted","Data":"536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51"}
Sep 16 21:36:52.869649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.869213    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz" event={"ID":"9948d9c9-87b8-4bc4-b696-0a36dc12578e","Type":"ContainerStarted","Data":"b0465c0339daff5526ca18c66c2f907d7c5c8e4566522e219b92bd52d6bf5a25"}
Sep 16 21:36:52.869649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.869267    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:36:52.871473 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.871434    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5" event={"ID":"0409a320-4439-4df8-bfda-25e80bb71cf9","Type":"ContainerStarted","Data":"46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2"}
Sep 16 21:36:52.871641 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.871625    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5" event={"ID":"0409a320-4439-4df8-bfda-25e80bb71cf9","Type":"ContainerStarted","Data":"ca563aff2f625b1ef0782481aa0374d29abbc8d44338ec32c7d4e11239c148e3"}
Sep 16 21:36:52.871780 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.871762    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:36:52.873138 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.873087    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb" event={"ID":"81065a73-1416-4f48-ae6a-9ca7ab954fb4","Type":"ContainerDied","Data":"7c38fdff22cd25fc833194cabc8d4ec037530b061990c290c852585e779debaf"}
Sep 16 21:36:52.873252 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.873184    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"
Sep 16 21:36:52.877999 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.877890    2567 scope.go:117] "RemoveContainer" containerID="45b30247e6198e66d66880a250660f8567af8f1a256d7fb969db387bdd0000bc"
Sep 16 21:36:52.890342 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.890296    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz" podStartSLOduration=1.890280916 podStartE2EDuration="1.890280916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:36:52.888031928 +0000 UTC m=+852.273715015" watchObservedRunningTime="2026-09-16 21:36:52.890280916 +0000 UTC m=+852.275964004"
Sep 16 21:36:52.905955 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.905919    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5" podStartSLOduration=1.905906262 podStartE2EDuration="1.905906262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:36:52.902963536 +0000 UTC m=+852.288646630" watchObservedRunningTime="2026-09-16 21:36:52.905906262 +0000 UTC m=+852.291589373"
Sep 16 21:36:52.925174 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.925150    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"]
Sep 16 21:36:52.929185 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.929156    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a1fb6d89-68df956b65-stkcb"]
Sep 16 21:36:52.937456 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.937432    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"]
Sep 16 21:36:52.941002 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:52.940982    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1fb6d89-6cf98fcd7d-t7gx5"]
Sep 16 21:36:53.109377 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:53.109341    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4abceba8-796f-4835-80b7-a7b48277795e" path="/var/lib/kubelet/pods/4abceba8-796f-4835-80b7-a7b48277795e/volumes"
Sep 16 21:36:53.109741 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:53.109726    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81065a73-1416-4f48-ae6a-9ca7ab954fb4" path="/var/lib/kubelet/pods/81065a73-1416-4f48-ae6a-9ca7ab954fb4/volumes"
Sep 16 21:36:53.110045 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:53.110033    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a425ffc4-95d6-48f8-82c4-cc564eeae8e6" path="/var/lib/kubelet/pods/a425ffc4-95d6-48f8-82c4-cc564eeae8e6/volumes"
Sep 16 21:36:59.126113 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:59.126082    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:59.126113 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:59.126124    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:59.139840 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:59.139810    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:36:59.935854 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:36:59.935826    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:37:01.583762 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:01.583730    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:37:03.882829 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:03.882798    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:37:04.890318 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:04.890286    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:37:17.990187 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:17.990151    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:37:35.676629 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:35.676548    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:37:44.189048 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:44.189014    2567 generic.go:356] "Generic (PLEG): container finished" podID="82aba2fe-2314-44ec-a8a8-404b6db3af87" containerID="bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb" exitCode=137
Sep 16 21:37:44.189508 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:44.189088    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm" event={"ID":"82aba2fe-2314-44ec-a8a8-404b6db3af87","Type":"ContainerDied","Data":"bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb"}
Sep 16 21:37:44.189588 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:44.189533    2567 scope.go:117] "RemoveContainer" containerID="bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb"
Sep 16 21:37:45.198081 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:45.198046    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm" event={"ID":"82aba2fe-2314-44ec-a8a8-404b6db3af87","Type":"ContainerStarted","Data":"7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a"}
Sep 16 21:37:45.198569 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:45.198264    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:37:51.591102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.591062    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"]
Sep 16 21:37:51.591624 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.591554    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a425ffc4-95d6-48f8-82c4-cc564eeae8e6" containerName="payload-pre-processing"
Sep 16 21:37:51.591624 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.591571    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a425ffc4-95d6-48f8-82c4-cc564eeae8e6" containerName="payload-pre-processing"
Sep 16 21:37:51.591624 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.591592    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81065a73-1416-4f48-ae6a-9ca7ab954fb4" containerName="payload-processing"
Sep 16 21:37:51.591624 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.591599    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="81065a73-1416-4f48-ae6a-9ca7ab954fb4" containerName="payload-processing"
Sep 16 21:37:51.591830 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.591678    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="81065a73-1416-4f48-ae6a-9ca7ab954fb4" containerName="payload-processing"
Sep 16 21:37:51.591830 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.591690    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a425ffc4-95d6-48f8-82c4-cc564eeae8e6" containerName="payload-pre-processing"
Sep 16 21:37:51.594850 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.594832    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.598205 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.598125    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b91829\"/\"explicit-model-b91829-kserve-self-signed-certs\""
Sep 16 21:37:51.598205 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.598134    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b91829\"/\"kube-root-ca.crt\""
Sep 16 21:37:51.598205 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.598193    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b91829\"/\"openshift-service-ca.crt\""
Sep 16 21:37:51.598469 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.598195    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b91829\"/\"default-dockercfg-srcvd\""
Sep 16 21:37:51.606212 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.606188    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"]
Sep 16 21:37:51.701024 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.700999    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tmp-dir\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.701137 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.701030    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvk8\" (UniqueName: \"kubernetes.io/projected/33e4bc8f-1039-4eb2-9551-46bfed20c50b-kube-api-access-fpvk8\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.701137 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.701054    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-model-cache\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.701137 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.701127    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tls-certs\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.701267 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.701225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-home\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.701267 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.701251    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-dshm\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.801839 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.801812    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-home\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.801839 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.801843    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-dshm\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.802038 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.801877    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tmp-dir\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.802038 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.801893    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvk8\" (UniqueName: \"kubernetes.io/projected/33e4bc8f-1039-4eb2-9551-46bfed20c50b-kube-api-access-fpvk8\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.802038 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.801914    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-model-cache\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.802038 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.801947    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tls-certs\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.802282 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.802257    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-home\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.802326 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.802270    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tmp-dir\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.802370 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.802350    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-model-cache\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.804270 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.804248    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-dshm\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.804502 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.804485    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tls-certs\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.808891 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.808873    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvk8\" (UniqueName: \"kubernetes.io/projected/33e4bc8f-1039-4eb2-9551-46bfed20c50b-kube-api-access-fpvk8\") pod \"explicit-model-b91829-kserve-6786ff5f47-2dqmx\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") " pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:51.907329 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:51.907257    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:37:52.178675 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:52.178651    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"]
Sep 16 21:37:52.180575 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:37:52.180539    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33e4bc8f_1039_4eb2_9551_46bfed20c50b.slice/crio-4924c8f28a4a217f6b4b97dd84e540ef75137e33c5dd71d995b9e9fece550b89 WatchSource:0}: Error finding container 4924c8f28a4a217f6b4b97dd84e540ef75137e33c5dd71d995b9e9fece550b89: Status 404 returned error can't find the container with id 4924c8f28a4a217f6b4b97dd84e540ef75137e33c5dd71d995b9e9fece550b89
Sep 16 21:37:52.241521 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:52.241495    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx" event={"ID":"33e4bc8f-1039-4eb2-9551-46bfed20c50b","Type":"ContainerStarted","Data":"4924c8f28a4a217f6b4b97dd84e540ef75137e33c5dd71d995b9e9fece550b89"}
Sep 16 21:37:53.256182 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:53.256146    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx" event={"ID":"33e4bc8f-1039-4eb2-9551-46bfed20c50b","Type":"ContainerStarted","Data":"9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a"}
Sep 16 21:37:53.273307 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:53.273260    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx" podStartSLOduration=2.273239011 podStartE2EDuration="2.273239011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:37:53.271560799 +0000 UTC m=+912.657243883" watchObservedRunningTime="2026-09-16 21:37:53.273239011 +0000 UTC m=+912.658922095"
Sep 16 21:37:56.019720 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.019686    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"]
Sep 16 21:37:56.146236 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.146198    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"]
Sep 16 21:37:56.150007 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.149991    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"
Sep 16 21:37:56.152335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.152310    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:37:56.153039 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.153008    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:37:56.159036 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.158690    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"]
Sep 16 21:37:56.205506 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.205487    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:37:56.239340 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.239311    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83276d69-d9d0-4afa-9f73-4a413d936cf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b91829-tx7pp\" (UID: \"83276d69-d9d0-4afa-9f73-4a413d936cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"
Sep 16 21:37:56.340793 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.340720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83276d69-d9d0-4afa-9f73-4a413d936cf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b91829-tx7pp\" (UID: \"83276d69-d9d0-4afa-9f73-4a413d936cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"
Sep 16 21:37:56.341283 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.341263    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83276d69-d9d0-4afa-9f73-4a413d936cf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b91829-tx7pp\" (UID: \"83276d69-d9d0-4afa-9f73-4a413d936cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"
Sep 16 21:37:56.462641 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.462616    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"
Sep 16 21:37:56.768354 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:37:56.768312    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83276d69_d9d0_4afa_9f73_4a413d936cf4.slice/crio-ae3d357c52554b1c34a1c1f5522140a2eaaaaf0660e30710dae4a1903627dd7b WatchSource:0}: Error finding container ae3d357c52554b1c34a1c1f5522140a2eaaaaf0660e30710dae4a1903627dd7b: Status 404 returned error can't find the container with id ae3d357c52554b1c34a1c1f5522140a2eaaaaf0660e30710dae4a1903627dd7b
Sep 16 21:37:56.773702 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:56.773672    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"]
Sep 16 21:37:57.280194 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:57.280154    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp" event={"ID":"83276d69-d9d0-4afa-9f73-4a413d936cf4","Type":"ContainerStarted","Data":"febf613a011642565f366712023bc3235f0fc088c2822a78eb5678cd33cc4eec"}
Sep 16 21:37:57.280675 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:57.280211    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp" event={"ID":"83276d69-d9d0-4afa-9f73-4a413d936cf4","Type":"ContainerStarted","Data":"ae3d357c52554b1c34a1c1f5522140a2eaaaaf0660e30710dae4a1903627dd7b"}
Sep 16 21:37:57.296117 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:57.296069    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp" podStartSLOduration=1.296055298 podStartE2EDuration="1.296055298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:37:57.293493005 +0000 UTC m=+916.679176103" watchObservedRunningTime="2026-09-16 21:37:57.296055298 +0000 UTC m=+916.681738382"
Sep 16 21:37:58.287828 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:58.287786    2567 generic.go:356] "Generic (PLEG): container finished" podID="83276d69-d9d0-4afa-9f73-4a413d936cf4" containerID="febf613a011642565f366712023bc3235f0fc088c2822a78eb5678cd33cc4eec" exitCode=0
Sep 16 21:37:58.288200 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:58.287847    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp" event={"ID":"83276d69-d9d0-4afa-9f73-4a413d936cf4","Type":"ContainerDied","Data":"febf613a011642565f366712023bc3235f0fc088c2822a78eb5678cd33cc4eec"}
Sep 16 21:37:59.432193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:59.432172    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"
Sep 16 21:37:59.573315 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:59.573220    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83276d69-d9d0-4afa-9f73-4a413d936cf4-maas-api-service-ca\") pod \"83276d69-d9d0-4afa-9f73-4a413d936cf4\" (UID: \"83276d69-d9d0-4afa-9f73-4a413d936cf4\") "
Sep 16 21:37:59.573603 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:59.573580    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83276d69-d9d0-4afa-9f73-4a413d936cf4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83276d69-d9d0-4afa-9f73-4a413d936cf4" (UID: "83276d69-d9d0-4afa-9f73-4a413d936cf4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:37:59.674736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:37:59.674709    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83276d69-d9d0-4afa-9f73-4a413d936cf4-maas-api-service-ca\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:38:00.301786 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:00.301754    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"
Sep 16 21:38:00.301955 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:00.301790    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp" event={"ID":"83276d69-d9d0-4afa-9f73-4a413d936cf4","Type":"ContainerDied","Data":"ae3d357c52554b1c34a1c1f5522140a2eaaaaf0660e30710dae4a1903627dd7b"}
Sep 16 21:38:00.301955 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:00.301838    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae3d357c52554b1c34a1c1f5522140a2eaaaaf0660e30710dae4a1903627dd7b"
Sep 16 21:38:01.907431 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:01.907381    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:38:06.728384 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:06.728343    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"]
Sep 16 21:38:06.729134 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:06.729100    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5" podUID="0409a320-4439-4df8-bfda-25e80bb71cf9" containerName="payload-processing" containerID="cri-o://46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2" gracePeriod=30
Sep 16 21:38:06.735246 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:06.733127    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"]
Sep 16 21:38:06.735246 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:06.735082    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz" podUID="9948d9c9-87b8-4bc4-b696-0a36dc12578e" containerName="payload-pre-processing" containerID="cri-o://536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51" gracePeriod=30
Sep 16 21:38:06.894574 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:06.894517    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"]
Sep 16 21:38:06.896937 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:06.896915    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91829-tx7pp"]
Sep 16 21:38:07.010464 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.010433    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:38:07.031598 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.031572    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:38:07.108353 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.108326    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83276d69-d9d0-4afa-9f73-4a413d936cf4" path="/var/lib/kubelet/pods/83276d69-d9d0-4afa-9f73-4a413d936cf4/volumes"
Sep 16 21:38:07.138917 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.138890    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0409a320-4439-4df8-bfda-25e80bb71cf9-plugins-config-volume\") pod \"0409a320-4439-4df8-bfda-25e80bb71cf9\" (UID: \"0409a320-4439-4df8-bfda-25e80bb71cf9\") "
Sep 16 21:38:07.139035 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.138933    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52grk\" (UniqueName: \"kubernetes.io/projected/0409a320-4439-4df8-bfda-25e80bb71cf9-kube-api-access-52grk\") pod \"0409a320-4439-4df8-bfda-25e80bb71cf9\" (UID: \"0409a320-4439-4df8-bfda-25e80bb71cf9\") "
Sep 16 21:38:07.139035 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.138965    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t84cl\" (UniqueName: \"kubernetes.io/projected/9948d9c9-87b8-4bc4-b696-0a36dc12578e-kube-api-access-t84cl\") pod \"9948d9c9-87b8-4bc4-b696-0a36dc12578e\" (UID: \"9948d9c9-87b8-4bc4-b696-0a36dc12578e\") "
Sep 16 21:38:07.139035 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.139032    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9948d9c9-87b8-4bc4-b696-0a36dc12578e-plugins-config-volume\") pod \"9948d9c9-87b8-4bc4-b696-0a36dc12578e\" (UID: \"9948d9c9-87b8-4bc4-b696-0a36dc12578e\") "
Sep 16 21:38:07.139337 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.139238    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0409a320-4439-4df8-bfda-25e80bb71cf9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0409a320-4439-4df8-bfda-25e80bb71cf9" (UID: "0409a320-4439-4df8-bfda-25e80bb71cf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:38:07.139472 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.139431    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9948d9c9-87b8-4bc4-b696-0a36dc12578e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9948d9c9-87b8-4bc4-b696-0a36dc12578e" (UID: "9948d9c9-87b8-4bc4-b696-0a36dc12578e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:38:07.141036 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.141016    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0409a320-4439-4df8-bfda-25e80bb71cf9-kube-api-access-52grk" (OuterVolumeSpecName: "kube-api-access-52grk") pod "0409a320-4439-4df8-bfda-25e80bb71cf9" (UID: "0409a320-4439-4df8-bfda-25e80bb71cf9"). InnerVolumeSpecName "kube-api-access-52grk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:38:07.141103 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.141031    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9948d9c9-87b8-4bc4-b696-0a36dc12578e-kube-api-access-t84cl" (OuterVolumeSpecName: "kube-api-access-t84cl") pod "9948d9c9-87b8-4bc4-b696-0a36dc12578e" (UID: "9948d9c9-87b8-4bc4-b696-0a36dc12578e"). InnerVolumeSpecName "kube-api-access-t84cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:38:07.239828 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.239760    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0409a320-4439-4df8-bfda-25e80bb71cf9-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:38:07.239828 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.239786    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52grk\" (UniqueName: \"kubernetes.io/projected/0409a320-4439-4df8-bfda-25e80bb71cf9-kube-api-access-52grk\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:38:07.239828 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.239801    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t84cl\" (UniqueName: \"kubernetes.io/projected/9948d9c9-87b8-4bc4-b696-0a36dc12578e-kube-api-access-t84cl\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:38:07.239828 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.239813    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9948d9c9-87b8-4bc4-b696-0a36dc12578e-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:38:07.344372 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.344343    2567 generic.go:356] "Generic (PLEG): container finished" podID="9948d9c9-87b8-4bc4-b696-0a36dc12578e" containerID="536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51" exitCode=0
Sep 16 21:38:07.344529 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.344386    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"
Sep 16 21:38:07.344529 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.344432    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz" event={"ID":"9948d9c9-87b8-4bc4-b696-0a36dc12578e","Type":"ContainerDied","Data":"536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51"}
Sep 16 21:38:07.344529 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.344488    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz" event={"ID":"9948d9c9-87b8-4bc4-b696-0a36dc12578e","Type":"ContainerDied","Data":"b0465c0339daff5526ca18c66c2f907d7c5c8e4566522e219b92bd52d6bf5a25"}
Sep 16 21:38:07.344529 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.344523    2567 scope.go:117] "RemoveContainer" containerID="536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51"
Sep 16 21:38:07.345955 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.345924    2567 generic.go:356] "Generic (PLEG): container finished" podID="0409a320-4439-4df8-bfda-25e80bb71cf9" containerID="46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2" exitCode=0
Sep 16 21:38:07.346060 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.345964    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"
Sep 16 21:38:07.346060 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.346014    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5" event={"ID":"0409a320-4439-4df8-bfda-25e80bb71cf9","Type":"ContainerDied","Data":"46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2"}
Sep 16 21:38:07.346140 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.346078    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5" event={"ID":"0409a320-4439-4df8-bfda-25e80bb71cf9","Type":"ContainerDied","Data":"ca563aff2f625b1ef0782481aa0374d29abbc8d44338ec32c7d4e11239c148e3"}
Sep 16 21:38:07.354524 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.354507    2567 scope.go:117] "RemoveContainer" containerID="536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51"
Sep 16 21:38:07.354901 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:38:07.354876    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51\": container with ID starting with 536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51 not found: ID does not exist" containerID="536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51"
Sep 16 21:38:07.354990 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.354906    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51"} err="failed to get container status \"536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51\": rpc error: code = NotFound desc = could not find container \"536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51\": container with ID starting with 536bc2bc911186170b492f0db75ff9689c02b8a14f05cc1fde7370d6cbb36a51 not found: ID does not exist"
Sep 16 21:38:07.354990 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.354923    2567 scope.go:117] "RemoveContainer" containerID="46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2"
Sep 16 21:38:07.364026 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.364004    2567 scope.go:117] "RemoveContainer" containerID="46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2"
Sep 16 21:38:07.364265 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:38:07.364243    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2\": container with ID starting with 46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2 not found: ID does not exist" containerID="46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2"
Sep 16 21:38:07.364325 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.364271    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2"} err="failed to get container status \"46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2\": rpc error: code = NotFound desc = could not find container \"46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2\": container with ID starting with 46fbc746e7953661f0f7dc6605aa41de497264749161cfb28769e6fbd3aa26d2 not found: ID does not exist"
Sep 16 21:38:07.368410 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.368378    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"]
Sep 16 21:38:07.373754 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.373734    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91829-547595bf78-6dbsz"]
Sep 16 21:38:07.385135 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.385113    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"]
Sep 16 21:38:07.388348 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:07.388326    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91829-6c5c668d97-2mfv5"]
Sep 16 21:38:09.111346 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:09.111297    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0409a320-4439-4df8-bfda-25e80bb71cf9" path="/var/lib/kubelet/pods/0409a320-4439-4df8-bfda-25e80bb71cf9/volumes"
Sep 16 21:38:09.111816 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:09.111682    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9948d9c9-87b8-4bc4-b696-0a36dc12578e" path="/var/lib/kubelet/pods/9948d9c9-87b8-4bc4-b696-0a36dc12578e/volumes"
Sep 16 21:38:09.448105 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:09.448078    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:38:11.042136 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.042093    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx" podUID="33e4bc8f-1039-4eb2-9551-46bfed20c50b" containerName="main" containerID="cri-o://9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a" gracePeriod=45
Sep 16 21:38:11.289936 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.289908    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:38:11.373560 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.373481    2567 generic.go:356] "Generic (PLEG): container finished" podID="33e4bc8f-1039-4eb2-9551-46bfed20c50b" containerID="9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a" exitCode=0
Sep 16 21:38:11.373560 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.373520    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"
Sep 16 21:38:11.373560 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.373538    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx" event={"ID":"33e4bc8f-1039-4eb2-9551-46bfed20c50b","Type":"ContainerDied","Data":"9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a"}
Sep 16 21:38:11.373771 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.373571    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx" event={"ID":"33e4bc8f-1039-4eb2-9551-46bfed20c50b","Type":"ContainerDied","Data":"4924c8f28a4a217f6b4b97dd84e540ef75137e33c5dd71d995b9e9fece550b89"}
Sep 16 21:38:11.373771 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.373591    2567 scope.go:117] "RemoveContainer" containerID="9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a"
Sep 16 21:38:11.382817 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.382795    2567 scope.go:117] "RemoveContainer" containerID="9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a"
Sep 16 21:38:11.383090 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:38:11.383070    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a\": container with ID starting with 9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a not found: ID does not exist" containerID="9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a"
Sep 16 21:38:11.383154 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.383102    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a"} err="failed to get container status \"9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a\": rpc error: code = NotFound desc = could not find container \"9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a\": container with ID starting with 9f906260ea1e5aba1e9f3ece00cf9dcb2ea2cdeb2bab0a2239ef8ff05f2d5c4a not found: ID does not exist"
Sep 16 21:38:11.478668 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.478634    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-model-cache\") pod \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") "
Sep 16 21:38:11.478819 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.478677    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpvk8\" (UniqueName: \"kubernetes.io/projected/33e4bc8f-1039-4eb2-9551-46bfed20c50b-kube-api-access-fpvk8\") pod \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") "
Sep 16 21:38:11.478819 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.478712    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tmp-dir\") pod \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") "
Sep 16 21:38:11.478819 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.478739    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tls-certs\") pod \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") "
Sep 16 21:38:11.478819 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.478770    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-dshm\") pod \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") "
Sep 16 21:38:11.478819 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.478810    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-home\") pod \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\" (UID: \"33e4bc8f-1039-4eb2-9551-46bfed20c50b\") "
Sep 16 21:38:11.479084 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.478935    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-model-cache" (OuterVolumeSpecName: "model-cache") pod "33e4bc8f-1039-4eb2-9551-46bfed20c50b" (UID: "33e4bc8f-1039-4eb2-9551-46bfed20c50b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:38:11.479139 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.479077    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-home" (OuterVolumeSpecName: "home") pod "33e4bc8f-1039-4eb2-9551-46bfed20c50b" (UID: "33e4bc8f-1039-4eb2-9551-46bfed20c50b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:38:11.479139 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.479129    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33e4bc8f-1039-4eb2-9551-46bfed20c50b" (UID: "33e4bc8f-1039-4eb2-9551-46bfed20c50b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:38:11.479248 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.479182    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-home\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:38:11.479248 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.479205    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-model-cache\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:38:11.481009 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.480979    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-dshm" (OuterVolumeSpecName: "dshm") pod "33e4bc8f-1039-4eb2-9551-46bfed20c50b" (UID: "33e4bc8f-1039-4eb2-9551-46bfed20c50b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:38:11.481104 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.481035    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33e4bc8f-1039-4eb2-9551-46bfed20c50b" (UID: "33e4bc8f-1039-4eb2-9551-46bfed20c50b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:38:11.481104 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.481040    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e4bc8f-1039-4eb2-9551-46bfed20c50b-kube-api-access-fpvk8" (OuterVolumeSpecName: "kube-api-access-fpvk8") pod "33e4bc8f-1039-4eb2-9551-46bfed20c50b" (UID: "33e4bc8f-1039-4eb2-9551-46bfed20c50b"). InnerVolumeSpecName "kube-api-access-fpvk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:38:11.580447 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.580389    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpvk8\" (UniqueName: \"kubernetes.io/projected/33e4bc8f-1039-4eb2-9551-46bfed20c50b-kube-api-access-fpvk8\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:38:11.580447 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.580443    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tmp-dir\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:38:11.580447 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.580453    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e4bc8f-1039-4eb2-9551-46bfed20c50b-tls-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:38:11.580660 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.580462    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e4bc8f-1039-4eb2-9551-46bfed20c50b-dshm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:38:11.696233 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.696201    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"]
Sep 16 21:38:11.701698 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:11.701671    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b91829/explicit-model-b91829-kserve-6786ff5f47-2dqmx"]
Sep 16 21:38:13.110529 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:13.110499    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e4bc8f-1039-4eb2-9551-46bfed20c50b" path="/var/lib/kubelet/pods/33e4bc8f-1039-4eb2-9551-46bfed20c50b/volumes"
Sep 16 21:38:21.968145 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968097    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"]
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968693    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9948d9c9-87b8-4bc4-b696-0a36dc12578e" containerName="payload-pre-processing"
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968712    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9948d9c9-87b8-4bc4-b696-0a36dc12578e" containerName="payload-pre-processing"
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968736    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83276d69-d9d0-4afa-9f73-4a413d936cf4" containerName="revoke-keys"
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968742    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="83276d69-d9d0-4afa-9f73-4a413d936cf4" containerName="revoke-keys"
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968766    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0409a320-4439-4df8-bfda-25e80bb71cf9" containerName="payload-processing"
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968774    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0409a320-4439-4df8-bfda-25e80bb71cf9" containerName="payload-processing"
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968783    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e4bc8f-1039-4eb2-9551-46bfed20c50b" containerName="main"
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968792    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e4bc8f-1039-4eb2-9551-46bfed20c50b" containerName="main"
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968876    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0409a320-4439-4df8-bfda-25e80bb71cf9" containerName="payload-processing"
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968885    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9948d9c9-87b8-4bc4-b696-0a36dc12578e" containerName="payload-pre-processing"
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968893    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e4bc8f-1039-4eb2-9551-46bfed20c50b" containerName="main"
Sep 16 21:38:21.970648 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.968899    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="83276d69-d9d0-4afa-9f73-4a413d936cf4" containerName="revoke-keys"
Sep 16 21:38:21.972009 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.971989    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:21.974517 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.974494    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"e2e-conflict-a-f2f38651-kserve-self-signed-certs\""
Sep 16 21:38:21.982142 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:21.982118    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"]
Sep 16 21:38:22.070290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.070253    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n874\" (UniqueName: \"kubernetes.io/projected/d1fc0d4c-0289-41ff-a701-6e4a50681111-kube-api-access-5n874\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.070290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.070289    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-dshm\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.070575 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.070318    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-tmp-dir\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.070575 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.070383    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-model-cache\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.070575 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.070452    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fc0d4c-0289-41ff-a701-6e4a50681111-tls-certs\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.070575 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.070480    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-home\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.170957 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.170922    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n874\" (UniqueName: \"kubernetes.io/projected/d1fc0d4c-0289-41ff-a701-6e4a50681111-kube-api-access-5n874\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.170957 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.170957    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-dshm\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.171188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.171004    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-tmp-dir\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.171188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.171031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-model-cache\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.171188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.171123    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fc0d4c-0289-41ff-a701-6e4a50681111-tls-certs\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.171188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.171165    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-home\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.171660 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.171634    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-model-cache\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.171821 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.171795    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-home\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.171901 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.171684    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-tmp-dir\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.173587 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.173562    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-dshm\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.173786 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.173771    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fc0d4c-0289-41ff-a701-6e4a50681111-tls-certs\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.178474 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.178456    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n874\" (UniqueName: \"kubernetes.io/projected/d1fc0d4c-0289-41ff-a701-6e4a50681111-kube-api-access-5n874\") pod \"e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.282709 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.282641    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:22.556173 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:22.556141    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"]
Sep 16 21:38:22.558023 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:38:22.558000    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1fc0d4c_0289_41ff_a701_6e4a50681111.slice/crio-772a6644f329bb17b2a7a0a80517083799bbe36a25ead6eb181f6ef05445175d WatchSource:0}: Error finding container 772a6644f329bb17b2a7a0a80517083799bbe36a25ead6eb181f6ef05445175d: Status 404 returned error can't find the container with id 772a6644f329bb17b2a7a0a80517083799bbe36a25ead6eb181f6ef05445175d
Sep 16 21:38:23.455080 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:23.455035    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww" event={"ID":"d1fc0d4c-0289-41ff-a701-6e4a50681111","Type":"ContainerStarted","Data":"4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa"}
Sep 16 21:38:23.455080 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:23.455085    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww" event={"ID":"d1fc0d4c-0289-41ff-a701-6e4a50681111","Type":"ContainerStarted","Data":"772a6644f329bb17b2a7a0a80517083799bbe36a25ead6eb181f6ef05445175d"}
Sep 16 21:38:23.472785 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:23.472707    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww" podStartSLOduration=2.472692142 podStartE2EDuration="2.472692142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:38:23.470830524 +0000 UTC m=+942.856513609" watchObservedRunningTime="2026-09-16 21:38:23.472692142 +0000 UTC m=+942.858375226"
Sep 16 21:38:32.283025 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:32.282992    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:32.283496 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:32.283039    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:32.295366 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:32.295343    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:32.524711 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:32.524683    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:38:36.076793 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:36.076759    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:38:37.916853 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:37.916823    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"]
Sep 16 21:38:37.920282 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:37.920267    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:38:37.922498 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:37.922470    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-06ac85db\""
Sep 16 21:38:37.931613 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:37.931587    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"]
Sep 16 21:38:37.958046 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:37.955948    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"]
Sep 16 21:38:37.961363 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:37.961340    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:38:37.964099 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:37.964076    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-06ac85db-dockercfg-2458s\""
Sep 16 21:38:37.968482 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:37.968456    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"]
Sep 16 21:38:38.017290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.017269    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcslg\" (UniqueName: \"kubernetes.io/projected/647bb6c8-0a0d-4353-a187-3370f8a471aa-kube-api-access-dcslg\") pod \"payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf\" (UID: \"647bb6c8-0a0d-4353-a187-3370f8a471aa\") " pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:38:38.017434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.017318    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647bb6c8-0a0d-4353-a187-3370f8a471aa-plugins-config-volume\") pod \"payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf\" (UID: \"647bb6c8-0a0d-4353-a187-3370f8a471aa\") " pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:38:38.017434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.017382    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6lbl\" (UniqueName: \"kubernetes.io/projected/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-kube-api-access-v6lbl\") pod \"payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv\" (UID: \"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:38:38.017514 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.017456    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv\" (UID: \"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:38:38.118272 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.118238    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcslg\" (UniqueName: \"kubernetes.io/projected/647bb6c8-0a0d-4353-a187-3370f8a471aa-kube-api-access-dcslg\") pod \"payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf\" (UID: \"647bb6c8-0a0d-4353-a187-3370f8a471aa\") " pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:38:38.118456 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.118292    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647bb6c8-0a0d-4353-a187-3370f8a471aa-plugins-config-volume\") pod \"payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf\" (UID: \"647bb6c8-0a0d-4353-a187-3370f8a471aa\") " pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:38:38.118456 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.118327    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6lbl\" (UniqueName: \"kubernetes.io/projected/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-kube-api-access-v6lbl\") pod \"payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv\" (UID: \"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:38:38.118456 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.118362    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv\" (UID: \"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:38:38.119071 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.119049    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647bb6c8-0a0d-4353-a187-3370f8a471aa-plugins-config-volume\") pod \"payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf\" (UID: \"647bb6c8-0a0d-4353-a187-3370f8a471aa\") " pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:38:38.119108 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.119058    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv\" (UID: \"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:38:38.134510 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.134480    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcslg\" (UniqueName: \"kubernetes.io/projected/647bb6c8-0a0d-4353-a187-3370f8a471aa-kube-api-access-dcslg\") pod \"payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf\" (UID: \"647bb6c8-0a0d-4353-a187-3370f8a471aa\") " pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:38:38.134624 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.134564    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6lbl\" (UniqueName: \"kubernetes.io/projected/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-kube-api-access-v6lbl\") pod \"payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv\" (UID: \"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:38:38.233389 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.233344    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:38:38.273270 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.273235    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:38:38.505921 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.505883    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"]
Sep 16 21:38:38.506917 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:38:38.506879    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9f38aa_7ade_4f48_9df4_3dac30dfaf60.slice/crio-339f6c0c36ecb90394e3ee7531d557c9c0387d472b78f79b3f2658d299d46ce8 WatchSource:0}: Error finding container 339f6c0c36ecb90394e3ee7531d557c9c0387d472b78f79b3f2658d299d46ce8: Status 404 returned error can't find the container with id 339f6c0c36ecb90394e3ee7531d557c9c0387d472b78f79b3f2658d299d46ce8
Sep 16 21:38:38.552841 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.552815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv" event={"ID":"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60","Type":"ContainerStarted","Data":"339f6c0c36ecb90394e3ee7531d557c9c0387d472b78f79b3f2658d299d46ce8"}
Sep 16 21:38:38.559091 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:38.559067    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"]
Sep 16 21:38:38.560071 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:38:38.560049    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647bb6c8_0a0d_4353_a187_3370f8a471aa.slice/crio-bd64a5093dc74ad61a804921db5fd9cc1057329c6cbb18a38487b9ab5d4d1715 WatchSource:0}: Error finding container bd64a5093dc74ad61a804921db5fd9cc1057329c6cbb18a38487b9ab5d4d1715: Status 404 returned error can't find the container with id bd64a5093dc74ad61a804921db5fd9cc1057329c6cbb18a38487b9ab5d4d1715
Sep 16 21:38:39.562326 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:39.562286    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf" event={"ID":"647bb6c8-0a0d-4353-a187-3370f8a471aa","Type":"ContainerStarted","Data":"eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd"}
Sep 16 21:38:39.562775 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:39.562354    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf" event={"ID":"647bb6c8-0a0d-4353-a187-3370f8a471aa","Type":"ContainerStarted","Data":"bd64a5093dc74ad61a804921db5fd9cc1057329c6cbb18a38487b9ab5d4d1715"}
Sep 16 21:38:39.562775 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:39.562501    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:38:39.564027 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:39.563997    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv" event={"ID":"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60","Type":"ContainerStarted","Data":"4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa"}
Sep 16 21:38:39.564145 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:39.564058    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:38:39.579473 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:39.579434    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf" podStartSLOduration=2.57942013 podStartE2EDuration="2.57942013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:38:39.57793768 +0000 UTC m=+958.963620789" watchObservedRunningTime="2026-09-16 21:38:39.57942013 +0000 UTC m=+958.965103216"
Sep 16 21:38:39.593975 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:39.593931    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv" podStartSLOduration=2.593916386 podStartE2EDuration="2.593916386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:38:39.59173379 +0000 UTC m=+958.977416900" watchObservedRunningTime="2026-09-16 21:38:39.593916386 +0000 UTC m=+958.979599507"
Sep 16 21:38:50.572586 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:50.572545    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:38:50.573071 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:38:50.572623    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:39:08.282032 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:08.281955    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:39:09.474865 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.474831    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"]
Sep 16 21:39:09.479379 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.479356    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.481990 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.481963    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-0a22a7\"/\"default-dockercfg-dx4jt\""
Sep 16 21:39:09.482764 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.482732    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-0a22a7\"/\"openshift-service-ca.crt\""
Sep 16 21:39:09.482764 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.482749    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-0a22a7\"/\"noten-model-0a22a7-kserve-self-signed-certs\""
Sep 16 21:39:09.482969 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.482789    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-0a22a7\"/\"kube-root-ca.crt\""
Sep 16 21:39:09.490419 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.490381    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"]
Sep 16 21:39:09.496062 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.496032    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbmrk\" (UniqueName: \"kubernetes.io/projected/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-kube-api-access-lbmrk\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.496190 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.496077    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-dshm\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.496259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.496197    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-home\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.496259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.496250    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-model-cache\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.496370 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.496287    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tmp-dir\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.496370 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.496310    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tls-certs\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.596823 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.596790    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-model-cache\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.596823 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.596824    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tmp-dir\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.597062 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.596839    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tls-certs\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.597062 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.596873    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbmrk\" (UniqueName: \"kubernetes.io/projected/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-kube-api-access-lbmrk\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.597062 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.596891    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-dshm\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.597062 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.597016    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-home\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.597299 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.597238    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-model-cache\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.597299 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.597264    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tmp-dir\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.597374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.597317    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-home\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.599111 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.599095    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-dshm\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.599356 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.599339    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tls-certs\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.604000 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.603981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbmrk\" (UniqueName: \"kubernetes.io/projected/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-kube-api-access-lbmrk\") pod \"noten-model-0a22a7-kserve-84c4dfff55-mqzc5\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") " pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:09.789931 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:09.789841    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:10.066894 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:10.066863    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"]
Sep 16 21:39:10.068095 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:39:10.068063    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod022a7697_6c58_4c3a_9e9e_b732fb7e1f6d.slice/crio-6dc2dfa2034068a415eaeae3830d53701099d5b661049b2a8fe3dd800bb05e90 WatchSource:0}: Error finding container 6dc2dfa2034068a415eaeae3830d53701099d5b661049b2a8fe3dd800bb05e90: Status 404 returned error can't find the container with id 6dc2dfa2034068a415eaeae3830d53701099d5b661049b2a8fe3dd800bb05e90
Sep 16 21:39:10.761765 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:10.761732    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5" event={"ID":"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d","Type":"ContainerStarted","Data":"d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07"}
Sep 16 21:39:10.762145 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:10.761778    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5" event={"ID":"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d","Type":"ContainerStarted","Data":"6dc2dfa2034068a415eaeae3830d53701099d5b661049b2a8fe3dd800bb05e90"}
Sep 16 21:39:10.779725 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:10.779675    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5" podStartSLOduration=1.779658586 podStartE2EDuration="1.779658586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:39:10.77673206 +0000 UTC m=+990.162415145" watchObservedRunningTime="2026-09-16 21:39:10.779658586 +0000 UTC m=+990.165341669"
Sep 16 21:39:19.790657 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:19.790616    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:19.791069 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:19.790666    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:19.802820 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:19.802797    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:19.828368 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:19.828318    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:21.882274 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:21.882227    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"]
Sep 16 21:39:21.882750 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:21.882478    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf" podUID="647bb6c8-0a0d-4353-a187-3370f8a471aa" containerName="payload-processing" containerID="cri-o://eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd" gracePeriod=30
Sep 16 21:39:21.887191 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:21.886909    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"]
Sep 16 21:39:21.887191 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:21.887120    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv" podUID="ec9f38aa-7ade-4f48-9df4-3dac30dfaf60" containerName="payload-pre-processing" containerID="cri-o://4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa" gracePeriod=30
Sep 16 21:39:22.148841 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.148818    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:39:22.177913 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.177892    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:39:22.206552 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.206525    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcslg\" (UniqueName: \"kubernetes.io/projected/647bb6c8-0a0d-4353-a187-3370f8a471aa-kube-api-access-dcslg\") pod \"647bb6c8-0a0d-4353-a187-3370f8a471aa\" (UID: \"647bb6c8-0a0d-4353-a187-3370f8a471aa\") "
Sep 16 21:39:22.206705 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.206566    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647bb6c8-0a0d-4353-a187-3370f8a471aa-plugins-config-volume\") pod \"647bb6c8-0a0d-4353-a187-3370f8a471aa\" (UID: \"647bb6c8-0a0d-4353-a187-3370f8a471aa\") "
Sep 16 21:39:22.207476 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.207451    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647bb6c8-0a0d-4353-a187-3370f8a471aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "647bb6c8-0a0d-4353-a187-3370f8a471aa" (UID: "647bb6c8-0a0d-4353-a187-3370f8a471aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:39:22.209363 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.209332    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647bb6c8-0a0d-4353-a187-3370f8a471aa-kube-api-access-dcslg" (OuterVolumeSpecName: "kube-api-access-dcslg") pod "647bb6c8-0a0d-4353-a187-3370f8a471aa" (UID: "647bb6c8-0a0d-4353-a187-3370f8a471aa"). InnerVolumeSpecName "kube-api-access-dcslg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:22.308139 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.308108    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6lbl\" (UniqueName: \"kubernetes.io/projected/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-kube-api-access-v6lbl\") pod \"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60\" (UID: \"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60\") "
Sep 16 21:39:22.308263 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.308187    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-plugins-config-volume\") pod \"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60\" (UID: \"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60\") "
Sep 16 21:39:22.308454 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.308435    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcslg\" (UniqueName: \"kubernetes.io/projected/647bb6c8-0a0d-4353-a187-3370f8a471aa-kube-api-access-dcslg\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:22.308499 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.308460    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647bb6c8-0a0d-4353-a187-3370f8a471aa-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:22.308594 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.308577    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec9f38aa-7ade-4f48-9df4-3dac30dfaf60" (UID: "ec9f38aa-7ade-4f48-9df4-3dac30dfaf60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:39:22.310092 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.310071    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-kube-api-access-v6lbl" (OuterVolumeSpecName: "kube-api-access-v6lbl") pod "ec9f38aa-7ade-4f48-9df4-3dac30dfaf60" (UID: "ec9f38aa-7ade-4f48-9df4-3dac30dfaf60"). InnerVolumeSpecName "kube-api-access-v6lbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:22.409421 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.409346    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6lbl\" (UniqueName: \"kubernetes.io/projected/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-kube-api-access-v6lbl\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:22.409421 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.409371    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:22.839125 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.839089    2567 generic.go:356] "Generic (PLEG): container finished" podID="647bb6c8-0a0d-4353-a187-3370f8a471aa" containerID="eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd" exitCode=0
Sep 16 21:39:22.839328 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.839140    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"
Sep 16 21:39:22.839328 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.839177    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf" event={"ID":"647bb6c8-0a0d-4353-a187-3370f8a471aa","Type":"ContainerDied","Data":"eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd"}
Sep 16 21:39:22.839328 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.839232    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf" event={"ID":"647bb6c8-0a0d-4353-a187-3370f8a471aa","Type":"ContainerDied","Data":"bd64a5093dc74ad61a804921db5fd9cc1057329c6cbb18a38487b9ab5d4d1715"}
Sep 16 21:39:22.839328 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.839251    2567 scope.go:117] "RemoveContainer" containerID="eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd"
Sep 16 21:39:22.840795 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.840766    2567 generic.go:356] "Generic (PLEG): container finished" podID="ec9f38aa-7ade-4f48-9df4-3dac30dfaf60" containerID="4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa" exitCode=0
Sep 16 21:39:22.840988 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.840802    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"
Sep 16 21:39:22.840988 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.840834    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv" event={"ID":"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60","Type":"ContainerDied","Data":"4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa"}
Sep 16 21:39:22.840988 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.840877    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv" event={"ID":"ec9f38aa-7ade-4f48-9df4-3dac30dfaf60","Type":"ContainerDied","Data":"339f6c0c36ecb90394e3ee7531d557c9c0387d472b78f79b3f2658d299d46ce8"}
Sep 16 21:39:22.849881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.849864    2567 scope.go:117] "RemoveContainer" containerID="eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd"
Sep 16 21:39:22.850228 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:39:22.850207    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd\": container with ID starting with eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd not found: ID does not exist" containerID="eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd"
Sep 16 21:39:22.850314 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.850237    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd"} err="failed to get container status \"eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd\": rpc error: code = NotFound desc = could not find container \"eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd\": container with ID starting with eeb427cad3f0c06cab6a755aa5bdd47ef8ccd8046d56deb4bcb448e1ee1dccbd not found: ID does not exist"
Sep 16 21:39:22.850314 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.850255    2567 scope.go:117] "RemoveContainer" containerID="4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa"
Sep 16 21:39:22.858625 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.858610    2567 scope.go:117] "RemoveContainer" containerID="4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa"
Sep 16 21:39:22.858863 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:39:22.858847    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa\": container with ID starting with 4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa not found: ID does not exist" containerID="4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa"
Sep 16 21:39:22.858906 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.858870    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa"} err="failed to get container status \"4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa\": rpc error: code = NotFound desc = could not find container \"4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa\": container with ID starting with 4ccdad17d45b54f452949812aba9f4427e82737baf9031b9276f13e6c3b9faaa not found: ID does not exist"
Sep 16 21:39:22.864202 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.864170    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"]
Sep 16 21:39:22.867998 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.867975    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06ac85db-85b74b7686-vvmzv"]
Sep 16 21:39:22.876546 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.876519    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"]
Sep 16 21:39:22.880386 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:22.880367    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06ac85db-76888b5b97-f8qtf"]
Sep 16 21:39:23.114259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:23.114177    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="647bb6c8-0a0d-4353-a187-3370f8a471aa" path="/var/lib/kubelet/pods/647bb6c8-0a0d-4353-a187-3370f8a471aa/volumes"
Sep 16 21:39:23.114625 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:23.114549    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec9f38aa-7ade-4f48-9df4-3dac30dfaf60" path="/var/lib/kubelet/pods/ec9f38aa-7ade-4f48-9df4-3dac30dfaf60/volumes"
Sep 16 21:39:27.881930 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:27.881892    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:39:41.691236 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:41.691147    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"]
Sep 16 21:39:44.003842 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.003808    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"]
Sep 16 21:39:44.004309 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.004233    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9f38aa-7ade-4f48-9df4-3dac30dfaf60" containerName="payload-pre-processing"
Sep 16 21:39:44.004309 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.004243    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9f38aa-7ade-4f48-9df4-3dac30dfaf60" containerName="payload-pre-processing"
Sep 16 21:39:44.004309 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.004272    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647bb6c8-0a0d-4353-a187-3370f8a471aa" containerName="payload-processing"
Sep 16 21:39:44.004309 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.004278    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="647bb6c8-0a0d-4353-a187-3370f8a471aa" containerName="payload-processing"
Sep 16 21:39:44.004570 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.004341    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec9f38aa-7ade-4f48-9df4-3dac30dfaf60" containerName="payload-pre-processing"
Sep 16 21:39:44.004570 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.004351    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="647bb6c8-0a0d-4353-a187-3370f8a471aa" containerName="payload-processing"
Sep 16 21:39:44.008724 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.008707    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"
Sep 16 21:39:44.011031 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.011008    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:39:44.011629 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.011605    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:39:44.014016 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.013986    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"]
Sep 16 21:39:44.114522 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.114490    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f74d1b-87ba-48e1-94fe-d5dce8292abb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4\" (UID: \"b7f74d1b-87ba-48e1-94fe-d5dce8292abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"
Sep 16 21:39:44.215629 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.215597    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f74d1b-87ba-48e1-94fe-d5dce8292abb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4\" (UID: \"b7f74d1b-87ba-48e1-94fe-d5dce8292abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"
Sep 16 21:39:44.216199 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.216178    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f74d1b-87ba-48e1-94fe-d5dce8292abb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4\" (UID: \"b7f74d1b-87ba-48e1-94fe-d5dce8292abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"
Sep 16 21:39:44.320728 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.320652    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"
Sep 16 21:39:44.590971 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.590938    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"]
Sep 16 21:39:44.591894 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:39:44.591862    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f74d1b_87ba_48e1_94fe_d5dce8292abb.slice/crio-bc1ac65292f988c548e45b2db12b747612e122b1d54b0ffdeddcb212ba5cda09 WatchSource:0}: Error finding container bc1ac65292f988c548e45b2db12b747612e122b1d54b0ffdeddcb212ba5cda09: Status 404 returned error can't find the container with id bc1ac65292f988c548e45b2db12b747612e122b1d54b0ffdeddcb212ba5cda09
Sep 16 21:39:44.985877 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.985850    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4" event={"ID":"b7f74d1b-87ba-48e1-94fe-d5dce8292abb","Type":"ContainerStarted","Data":"109b6abaabbb4c5b407d9b9cf2b842c8ebf535666827ac6e867e3d78a5c887fa"}
Sep 16 21:39:44.986009 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:44.985898    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4" event={"ID":"b7f74d1b-87ba-48e1-94fe-d5dce8292abb","Type":"ContainerStarted","Data":"bc1ac65292f988c548e45b2db12b747612e122b1d54b0ffdeddcb212ba5cda09"}
Sep 16 21:39:45.001703 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:45.001661    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4" podStartSLOduration=2.001646978 podStartE2EDuration="2.001646978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:39:44.999572424 +0000 UTC m=+1024.385255507" watchObservedRunningTime="2026-09-16 21:39:45.001646978 +0000 UTC m=+1024.387330062"
Sep 16 21:39:45.993462 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:45.993430    2567 generic.go:356] "Generic (PLEG): container finished" podID="b7f74d1b-87ba-48e1-94fe-d5dce8292abb" containerID="109b6abaabbb4c5b407d9b9cf2b842c8ebf535666827ac6e867e3d78a5c887fa" exitCode=0
Sep 16 21:39:45.993864 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:45.993522    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4" event={"ID":"b7f74d1b-87ba-48e1-94fe-d5dce8292abb","Type":"ContainerDied","Data":"109b6abaabbb4c5b407d9b9cf2b842c8ebf535666827ac6e867e3d78a5c887fa"}
Sep 16 21:39:47.138101 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:47.138075    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"
Sep 16 21:39:47.140452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:47.140428    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f74d1b-87ba-48e1-94fe-d5dce8292abb-maas-api-service-ca\") pod \"b7f74d1b-87ba-48e1-94fe-d5dce8292abb\" (UID: \"b7f74d1b-87ba-48e1-94fe-d5dce8292abb\") "
Sep 16 21:39:47.140886 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:47.140840    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f74d1b-87ba-48e1-94fe-d5dce8292abb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7f74d1b-87ba-48e1-94fe-d5dce8292abb" (UID: "b7f74d1b-87ba-48e1-94fe-d5dce8292abb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:39:47.241339 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:47.241304    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f74d1b-87ba-48e1-94fe-d5dce8292abb-maas-api-service-ca\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:48.006121 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:48.006091    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"
Sep 16 21:39:48.006328 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:48.006125    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4" event={"ID":"b7f74d1b-87ba-48e1-94fe-d5dce8292abb","Type":"ContainerDied","Data":"bc1ac65292f988c548e45b2db12b747612e122b1d54b0ffdeddcb212ba5cda09"}
Sep 16 21:39:48.006328 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:48.006183    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc1ac65292f988c548e45b2db12b747612e122b1d54b0ffdeddcb212ba5cda09"
Sep 16 21:39:49.750615 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.750580    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"]
Sep 16 21:39:49.751023 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.751010    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7f74d1b-87ba-48e1-94fe-d5dce8292abb" containerName="revoke-keys"
Sep 16 21:39:49.751023 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.751024    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7f74d1b-87ba-48e1-94fe-d5dce8292abb" containerName="revoke-keys"
Sep 16 21:39:49.751101 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.751094    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7f74d1b-87ba-48e1-94fe-d5dce8292abb" containerName="revoke-keys"
Sep 16 21:39:49.754065 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.754048    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:39:49.756188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.756167    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-296002\""
Sep 16 21:39:49.756332 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.756213    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-296002-dockercfg-h26m6\""
Sep 16 21:39:49.763140 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.763120    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm8x9\" (UniqueName: \"kubernetes.io/projected/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-kube-api-access-tm8x9\") pod \"payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx\" (UID: \"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:39:49.763288 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.763177    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx\" (UID: \"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:39:49.765598 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.765577    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"]
Sep 16 21:39:49.786322 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.786298    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"]
Sep 16 21:39:49.789578 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.789562    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:39:49.805202 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.805182    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"]
Sep 16 21:39:49.864359 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.864329    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p742b\" (UniqueName: \"kubernetes.io/projected/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-kube-api-access-p742b\") pod \"payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg\" (UID: \"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:39:49.864521 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.864448    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm8x9\" (UniqueName: \"kubernetes.io/projected/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-kube-api-access-tm8x9\") pod \"payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx\" (UID: \"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:39:49.864521 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.864491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx\" (UID: \"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:39:49.864602 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.864550    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg\" (UID: \"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:39:49.865146 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.865125    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx\" (UID: \"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:39:49.871856 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.871834    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm8x9\" (UniqueName: \"kubernetes.io/projected/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-kube-api-access-tm8x9\") pod \"payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx\" (UID: \"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:39:49.965120 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.965089    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p742b\" (UniqueName: \"kubernetes.io/projected/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-kube-api-access-p742b\") pod \"payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg\" (UID: \"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:39:49.965246 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.965220    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg\" (UID: \"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:39:49.965938 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.965919    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg\" (UID: \"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:39:49.972493 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:49.972466    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p742b\" (UniqueName: \"kubernetes.io/projected/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-kube-api-access-p742b\") pod \"payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg\" (UID: \"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:39:50.064841 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:50.064777    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:39:50.100208 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:50.100171    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:39:50.342213 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:50.342181    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"]
Sep 16 21:39:50.344195 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:39:50.344167    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b5ef76_65e6_4f9e_a7ee_d4b079644bfa.slice/crio-752b32739cc348e14f008c6d52d51e6f5d63d799e97e9293dcfac96018b716dd WatchSource:0}: Error finding container 752b32739cc348e14f008c6d52d51e6f5d63d799e97e9293dcfac96018b716dd: Status 404 returned error can't find the container with id 752b32739cc348e14f008c6d52d51e6f5d63d799e97e9293dcfac96018b716dd
Sep 16 21:39:50.391668 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:50.391645    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"]
Sep 16 21:39:50.394569 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:39:50.394542    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1b7dcd_d05f_4449_9618_1bd9f23a6e0a.slice/crio-a68082b8bdc50f46cbb69c16380e2703db1f9e264bbc89d2458b85c67bf1c6a1 WatchSource:0}: Error finding container a68082b8bdc50f46cbb69c16380e2703db1f9e264bbc89d2458b85c67bf1c6a1: Status 404 returned error can't find the container with id a68082b8bdc50f46cbb69c16380e2703db1f9e264bbc89d2458b85c67bf1c6a1
Sep 16 21:39:51.035050 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:51.035006    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg" event={"ID":"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a","Type":"ContainerStarted","Data":"40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e"}
Sep 16 21:39:51.035616 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:51.035108    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg" event={"ID":"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a","Type":"ContainerStarted","Data":"a68082b8bdc50f46cbb69c16380e2703db1f9e264bbc89d2458b85c67bf1c6a1"}
Sep 16 21:39:51.035616 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:51.035303    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:39:51.037033 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:51.037007    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx" event={"ID":"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa","Type":"ContainerStarted","Data":"505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b"}
Sep 16 21:39:51.037161 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:51.037050    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx" event={"ID":"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa","Type":"ContainerStarted","Data":"752b32739cc348e14f008c6d52d51e6f5d63d799e97e9293dcfac96018b716dd"}
Sep 16 21:39:51.037161 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:51.037109    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:39:51.053889 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:51.053846    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg" podStartSLOduration=2.053834068 podStartE2EDuration="2.053834068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:39:51.052695285 +0000 UTC m=+1030.438378388" watchObservedRunningTime="2026-09-16 21:39:51.053834068 +0000 UTC m=+1030.439517221"
Sep 16 21:39:51.074957 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:51.074914    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx" podStartSLOduration=2.074902077 podStartE2EDuration="2.074902077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:39:51.073147461 +0000 UTC m=+1030.458830561" watchObservedRunningTime="2026-09-16 21:39:51.074902077 +0000 UTC m=+1030.460585160"
Sep 16 21:39:54.730287 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:54.730253    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"]
Sep 16 21:39:54.731035 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:54.731002    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm" podUID="80a4fe71-7147-4091-831e-adc9a450c576" containerName="payload-processing" containerID="cri-o://66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446" gracePeriod=30
Sep 16 21:39:54.734719 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:54.734693    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"]
Sep 16 21:39:54.735050 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:54.735003    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8" podUID="1d0a6186-19c9-4c74-8082-999432b88093" containerName="payload-pre-processing" containerID="cri-o://d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6" gracePeriod=30
Sep 16 21:39:54.971561 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:54.971527    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"]
Sep 16 21:39:54.974028 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:54.974003    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5568d-mg6q4"]
Sep 16 21:39:55.019488 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.019463    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:39:55.024092 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.024072    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:39:55.072058 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.072015    2567 generic.go:356] "Generic (PLEG): container finished" podID="80a4fe71-7147-4091-831e-adc9a450c576" containerID="66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446" exitCode=0
Sep 16 21:39:55.072236 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.072068    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"
Sep 16 21:39:55.072236 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.072074    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm" event={"ID":"80a4fe71-7147-4091-831e-adc9a450c576","Type":"ContainerDied","Data":"66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446"}
Sep 16 21:39:55.072236 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.072135    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm" event={"ID":"80a4fe71-7147-4091-831e-adc9a450c576","Type":"ContainerDied","Data":"a6d6b5add5ec8f9b1c0eeadd174cccb9a875820dda3f7287b3f257270d4de8f4"}
Sep 16 21:39:55.072236 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.072165    2567 scope.go:117] "RemoveContainer" containerID="66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446"
Sep 16 21:39:55.074187 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.073992    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d0a6186-19c9-4c74-8082-999432b88093" containerID="d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6" exitCode=0
Sep 16 21:39:55.074187 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.074107    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8" event={"ID":"1d0a6186-19c9-4c74-8082-999432b88093","Type":"ContainerDied","Data":"d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6"}
Sep 16 21:39:55.074187 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.074140    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"
Sep 16 21:39:55.074187 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.074147    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8" event={"ID":"1d0a6186-19c9-4c74-8082-999432b88093","Type":"ContainerDied","Data":"f596f0798b2537d6c007f57fc8ddb8fcffe4214ed17754537b7b9f36492a5501"}
Sep 16 21:39:55.083979 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.083956    2567 scope.go:117] "RemoveContainer" containerID="66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446"
Sep 16 21:39:55.084330 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:39:55.084287    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446\": container with ID starting with 66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446 not found: ID does not exist" containerID="66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446"
Sep 16 21:39:55.084492 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.084323    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446"} err="failed to get container status \"66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446\": rpc error: code = NotFound desc = could not find container \"66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446\": container with ID starting with 66998cf1c81b4388042dddc5fc915ae5e39a2a3aac9749f7bc3601501197f446 not found: ID does not exist"
Sep 16 21:39:55.084492 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.084348    2567 scope.go:117] "RemoveContainer" containerID="d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6"
Sep 16 21:39:55.093239 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.093221    2567 scope.go:117] "RemoveContainer" containerID="d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6"
Sep 16 21:39:55.093516 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:39:55.093498    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6\": container with ID starting with d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6 not found: ID does not exist" containerID="d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6"
Sep 16 21:39:55.093593 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.093527    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6"} err="failed to get container status \"d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6\": rpc error: code = NotFound desc = could not find container \"d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6\": container with ID starting with d4480b42825a012d777922fab00eed04768b0718bb9ad0e3e70319644c632ff6 not found: ID does not exist"
Sep 16 21:39:55.109117 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.109094    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7f74d1b-87ba-48e1-94fe-d5dce8292abb" path="/var/lib/kubelet/pods/b7f74d1b-87ba-48e1-94fe-d5dce8292abb/volumes"
Sep 16 21:39:55.113552 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.113532    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0a6186-19c9-4c74-8082-999432b88093-plugins-config-volume\") pod \"1d0a6186-19c9-4c74-8082-999432b88093\" (UID: \"1d0a6186-19c9-4c74-8082-999432b88093\") "
Sep 16 21:39:55.113635 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.113570    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h47vx\" (UniqueName: \"kubernetes.io/projected/80a4fe71-7147-4091-831e-adc9a450c576-kube-api-access-h47vx\") pod \"80a4fe71-7147-4091-831e-adc9a450c576\" (UID: \"80a4fe71-7147-4091-831e-adc9a450c576\") "
Sep 16 21:39:55.113635 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.113591    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7j66\" (UniqueName: \"kubernetes.io/projected/1d0a6186-19c9-4c74-8082-999432b88093-kube-api-access-r7j66\") pod \"1d0a6186-19c9-4c74-8082-999432b88093\" (UID: \"1d0a6186-19c9-4c74-8082-999432b88093\") "
Sep 16 21:39:55.113703 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.113683    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a4fe71-7147-4091-831e-adc9a450c576-plugins-config-volume\") pod \"80a4fe71-7147-4091-831e-adc9a450c576\" (UID: \"80a4fe71-7147-4091-831e-adc9a450c576\") "
Sep 16 21:39:55.113914 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.113861    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0a6186-19c9-4c74-8082-999432b88093-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d0a6186-19c9-4c74-8082-999432b88093" (UID: "1d0a6186-19c9-4c74-8082-999432b88093"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:39:55.114042 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.114023    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a4fe71-7147-4091-831e-adc9a450c576-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80a4fe71-7147-4091-831e-adc9a450c576" (UID: "80a4fe71-7147-4091-831e-adc9a450c576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:39:55.115609 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.115586    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a4fe71-7147-4091-831e-adc9a450c576-kube-api-access-h47vx" (OuterVolumeSpecName: "kube-api-access-h47vx") pod "80a4fe71-7147-4091-831e-adc9a450c576" (UID: "80a4fe71-7147-4091-831e-adc9a450c576"). InnerVolumeSpecName "kube-api-access-h47vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:55.115717 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.115700    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0a6186-19c9-4c74-8082-999432b88093-kube-api-access-r7j66" (OuterVolumeSpecName: "kube-api-access-r7j66") pod "1d0a6186-19c9-4c74-8082-999432b88093" (UID: "1d0a6186-19c9-4c74-8082-999432b88093"). InnerVolumeSpecName "kube-api-access-r7j66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:55.215204 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.215177    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h47vx\" (UniqueName: \"kubernetes.io/projected/80a4fe71-7147-4091-831e-adc9a450c576-kube-api-access-h47vx\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.215204 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.215200    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7j66\" (UniqueName: \"kubernetes.io/projected/1d0a6186-19c9-4c74-8082-999432b88093-kube-api-access-r7j66\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.215344 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.215210    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a4fe71-7147-4091-831e-adc9a450c576-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.215344 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.215220    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0a6186-19c9-4c74-8082-999432b88093-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.395159 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.395126    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"]
Sep 16 21:39:55.404453 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.404422    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e5568d-566c97bb8c-8tfpm"]
Sep 16 21:39:55.418302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.418276    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"]
Sep 16 21:39:55.425755 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:55.425729    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e5568d-678b8b9dbb-gbgv8"]
Sep 16 21:39:56.712483 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:56.712445    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5" podUID="022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" containerName="main" containerID="cri-o://d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07" gracePeriod=45
Sep 16 21:39:56.952752 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:56.952726    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:57.035686 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.035607    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-dshm\") pod \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") "
Sep 16 21:39:57.035686 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.035656    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tmp-dir\") pod \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") "
Sep 16 21:39:57.035686 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.035676    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tls-certs\") pod \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") "
Sep 16 21:39:57.035969 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.035726    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbmrk\" (UniqueName: \"kubernetes.io/projected/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-kube-api-access-lbmrk\") pod \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") "
Sep 16 21:39:57.035969 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.035785    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-model-cache\") pod \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") "
Sep 16 21:39:57.035969 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.035847    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-home\") pod \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\" (UID: \"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d\") "
Sep 16 21:39:57.035969 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.035877    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" (UID: "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:57.036200 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.036079    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" (UID: "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:57.036200 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.036143    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-home" (OuterVolumeSpecName: "home") pod "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" (UID: "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:57.036200 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.036167    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tmp-dir\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:57.036200 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.036186    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-model-cache\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:57.038051 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.038021    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-kube-api-access-lbmrk" (OuterVolumeSpecName: "kube-api-access-lbmrk") pod "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" (UID: "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d"). InnerVolumeSpecName "kube-api-access-lbmrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:57.038177 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.038114    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" (UID: "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:39:57.038177 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.038124    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-dshm" (OuterVolumeSpecName: "dshm") pod "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" (UID: "022a7697-6c58-4c3a-9e9e-b732fb7e1f6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:57.089646 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.089613    2567 generic.go:356] "Generic (PLEG): container finished" podID="022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" containerID="d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07" exitCode=0
Sep 16 21:39:57.089821 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.089695    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"
Sep 16 21:39:57.089821 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.089700    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5" event={"ID":"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d","Type":"ContainerDied","Data":"d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07"}
Sep 16 21:39:57.089821 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.089747    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5" event={"ID":"022a7697-6c58-4c3a-9e9e-b732fb7e1f6d","Type":"ContainerDied","Data":"6dc2dfa2034068a415eaeae3830d53701099d5b661049b2a8fe3dd800bb05e90"}
Sep 16 21:39:57.089821 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.089764    2567 scope.go:117] "RemoveContainer" containerID="d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07"
Sep 16 21:39:57.099520 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.099503    2567 scope.go:117] "RemoveContainer" containerID="d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07"
Sep 16 21:39:57.099766 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:39:57.099745    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07\": container with ID starting with d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07 not found: ID does not exist" containerID="d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07"
Sep 16 21:39:57.099811 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.099775    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07"} err="failed to get container status \"d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07\": rpc error: code = NotFound desc = could not find container \"d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07\": container with ID starting with d063975e68cb75f5dae2d6cfe9144abba8740d4c0ab18489e0717d0a3c997c07 not found: ID does not exist"
Sep 16 21:39:57.108638 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.108618    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d0a6186-19c9-4c74-8082-999432b88093" path="/var/lib/kubelet/pods/1d0a6186-19c9-4c74-8082-999432b88093/volumes"
Sep 16 21:39:57.108952 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.108939    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80a4fe71-7147-4091-831e-adc9a450c576" path="/var/lib/kubelet/pods/80a4fe71-7147-4091-831e-adc9a450c576/volumes"
Sep 16 21:39:57.116000 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.115978    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"]
Sep 16 21:39:57.118558 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.118540    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-0a22a7/noten-model-0a22a7-kserve-84c4dfff55-mqzc5"]
Sep 16 21:39:57.136991 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.136966    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbmrk\" (UniqueName: \"kubernetes.io/projected/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-kube-api-access-lbmrk\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:57.136991 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.136990    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-home\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:57.137219 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.137005    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-dshm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:57.137219 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:57.137017    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d-tls-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:39:59.109363 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:39:59.109330    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" path="/var/lib/kubelet/pods/022a7697-6c58-4c3a-9e9e-b732fb7e1f6d/volumes"
Sep 16 21:40:00.099266 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:00.099229    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:40:02.045986 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:02.045950    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:40:02.046460 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:02.046039    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:40:13.288196 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.288157    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"]
Sep 16 21:40:13.288720 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.288376    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh" podUID="7e25bec0-1e2a-486e-832f-c76ead1393ae" containerName="payload-processing" containerID="cri-o://e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a" gracePeriod=30
Sep 16 21:40:13.300359 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.300331    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"]
Sep 16 21:40:13.300616 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.300590    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j" podUID="122898b2-56eb-4c82-9036-ddf9f6a246e4" containerName="payload-pre-processing" containerID="cri-o://2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290" gracePeriod=30
Sep 16 21:40:13.575959 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.575934    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:40:13.583972 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.583952    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:40:13.689891 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.689860    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjbdz\" (UniqueName: \"kubernetes.io/projected/7e25bec0-1e2a-486e-832f-c76ead1393ae-kube-api-access-cjbdz\") pod \"7e25bec0-1e2a-486e-832f-c76ead1393ae\" (UID: \"7e25bec0-1e2a-486e-832f-c76ead1393ae\") "
Sep 16 21:40:13.690078 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.689914    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122898b2-56eb-4c82-9036-ddf9f6a246e4-plugins-config-volume\") pod \"122898b2-56eb-4c82-9036-ddf9f6a246e4\" (UID: \"122898b2-56eb-4c82-9036-ddf9f6a246e4\") "
Sep 16 21:40:13.690078 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.689943    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e25bec0-1e2a-486e-832f-c76ead1393ae-plugins-config-volume\") pod \"7e25bec0-1e2a-486e-832f-c76ead1393ae\" (UID: \"7e25bec0-1e2a-486e-832f-c76ead1393ae\") "
Sep 16 21:40:13.690204 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.690118    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r224l\" (UniqueName: \"kubernetes.io/projected/122898b2-56eb-4c82-9036-ddf9f6a246e4-kube-api-access-r224l\") pod \"122898b2-56eb-4c82-9036-ddf9f6a246e4\" (UID: \"122898b2-56eb-4c82-9036-ddf9f6a246e4\") "
Sep 16 21:40:13.690264 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.690254    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e25bec0-1e2a-486e-832f-c76ead1393ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e25bec0-1e2a-486e-832f-c76ead1393ae" (UID: "7e25bec0-1e2a-486e-832f-c76ead1393ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:40:13.690322 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.690285    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122898b2-56eb-4c82-9036-ddf9f6a246e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "122898b2-56eb-4c82-9036-ddf9f6a246e4" (UID: "122898b2-56eb-4c82-9036-ddf9f6a246e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:40:13.690507 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.690489    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122898b2-56eb-4c82-9036-ddf9f6a246e4-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:40:13.690577 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.690508    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e25bec0-1e2a-486e-832f-c76ead1393ae-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:40:13.691920 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.691895    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e25bec0-1e2a-486e-832f-c76ead1393ae-kube-api-access-cjbdz" (OuterVolumeSpecName: "kube-api-access-cjbdz") pod "7e25bec0-1e2a-486e-832f-c76ead1393ae" (UID: "7e25bec0-1e2a-486e-832f-c76ead1393ae"). InnerVolumeSpecName "kube-api-access-cjbdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:40:13.692033 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.692016    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122898b2-56eb-4c82-9036-ddf9f6a246e4-kube-api-access-r224l" (OuterVolumeSpecName: "kube-api-access-r224l") pod "122898b2-56eb-4c82-9036-ddf9f6a246e4" (UID: "122898b2-56eb-4c82-9036-ddf9f6a246e4"). InnerVolumeSpecName "kube-api-access-r224l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:40:13.791204 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.791174    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r224l\" (UniqueName: \"kubernetes.io/projected/122898b2-56eb-4c82-9036-ddf9f6a246e4-kube-api-access-r224l\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:40:13.791204 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:13.791201    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjbdz\" (UniqueName: \"kubernetes.io/projected/7e25bec0-1e2a-486e-832f-c76ead1393ae-kube-api-access-cjbdz\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:40:14.207102 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.207066    2567 generic.go:356] "Generic (PLEG): container finished" podID="7e25bec0-1e2a-486e-832f-c76ead1393ae" containerID="e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a" exitCode=0
Sep 16 21:40:14.207260 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.207107    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"
Sep 16 21:40:14.207260 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.207150    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh" event={"ID":"7e25bec0-1e2a-486e-832f-c76ead1393ae","Type":"ContainerDied","Data":"e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a"}
Sep 16 21:40:14.207260 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.207211    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh" event={"ID":"7e25bec0-1e2a-486e-832f-c76ead1393ae","Type":"ContainerDied","Data":"234bccc85ff75d5a99dcb6010e8b8cdc81a4dcd44383e0127fca30e0f969a64e"}
Sep 16 21:40:14.207260 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.207245    2567 scope.go:117] "RemoveContainer" containerID="e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a"
Sep 16 21:40:14.208596 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.208573    2567 generic.go:356] "Generic (PLEG): container finished" podID="122898b2-56eb-4c82-9036-ddf9f6a246e4" containerID="2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290" exitCode=0
Sep 16 21:40:14.208730 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.208609    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j" event={"ID":"122898b2-56eb-4c82-9036-ddf9f6a246e4","Type":"ContainerDied","Data":"2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290"}
Sep 16 21:40:14.208730 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.208641    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"
Sep 16 21:40:14.208730 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.208645    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j" event={"ID":"122898b2-56eb-4c82-9036-ddf9f6a246e4","Type":"ContainerDied","Data":"c53f14ccbc1d3f2391b4610c999f5c133104f57eb642b7c25a698e0f89e772cb"}
Sep 16 21:40:14.222106 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.218258    2567 scope.go:117] "RemoveContainer" containerID="e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a"
Sep 16 21:40:14.222106 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:40:14.218609    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a\": container with ID starting with e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a not found: ID does not exist" containerID="e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a"
Sep 16 21:40:14.222106 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.218650    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a"} err="failed to get container status \"e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a\": rpc error: code = NotFound desc = could not find container \"e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a\": container with ID starting with e5caf0014977b50b9404f9c01b75095729afb8e62ae187837afe7033c874897a not found: ID does not exist"
Sep 16 21:40:14.222106 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.218673    2567 scope.go:117] "RemoveContainer" containerID="2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290"
Sep 16 21:40:14.232315 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.232301    2567 scope.go:117] "RemoveContainer" containerID="2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290"
Sep 16 21:40:14.232634 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:40:14.232615    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290\": container with ID starting with 2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290 not found: ID does not exist" containerID="2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290"
Sep 16 21:40:14.232694 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.232643    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290"} err="failed to get container status \"2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290\": rpc error: code = NotFound desc = could not find container \"2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290\": container with ID starting with 2a5aeae26ef93f1bd9aa056891583e5e636d83410b9ac25e581f1e1dbef7d290 not found: ID does not exist"
Sep 16 21:40:14.239184 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.239162    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"]
Sep 16 21:40:14.244649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.244626    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44e517-666645dd94-msl9j"]
Sep 16 21:40:14.253325 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.253306    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"]
Sep 16 21:40:14.257188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:14.257169    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44e517-6f764db755-sshjh"]
Sep 16 21:40:15.109065 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:15.109033    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="122898b2-56eb-4c82-9036-ddf9f6a246e4" path="/var/lib/kubelet/pods/122898b2-56eb-4c82-9036-ddf9f6a246e4/volumes"
Sep 16 21:40:15.109454 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:15.109413    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e25bec0-1e2a-486e-832f-c76ead1393ae" path="/var/lib/kubelet/pods/7e25bec0-1e2a-486e-832f-c76ead1393ae/volumes"
Sep 16 21:40:16.264021 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.263988    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"]
Sep 16 21:40:16.264432 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264391    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a4fe71-7147-4091-831e-adc9a450c576" containerName="payload-processing"
Sep 16 21:40:16.264432 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264432    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a4fe71-7147-4091-831e-adc9a450c576" containerName="payload-processing"
Sep 16 21:40:16.264531 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264445    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d0a6186-19c9-4c74-8082-999432b88093" containerName="payload-pre-processing"
Sep 16 21:40:16.264531 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264451    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d0a6186-19c9-4c74-8082-999432b88093" containerName="payload-pre-processing"
Sep 16 21:40:16.264531 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264457    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e25bec0-1e2a-486e-832f-c76ead1393ae" containerName="payload-processing"
Sep 16 21:40:16.264531 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264462    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e25bec0-1e2a-486e-832f-c76ead1393ae" containerName="payload-processing"
Sep 16 21:40:16.264531 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264470    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" containerName="main"
Sep 16 21:40:16.264531 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264476    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" containerName="main"
Sep 16 21:40:16.264531 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264497    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122898b2-56eb-4c82-9036-ddf9f6a246e4" containerName="payload-pre-processing"
Sep 16 21:40:16.264531 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264502    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="122898b2-56eb-4c82-9036-ddf9f6a246e4" containerName="payload-pre-processing"
Sep 16 21:40:16.264767 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264565    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e25bec0-1e2a-486e-832f-c76ead1393ae" containerName="payload-processing"
Sep 16 21:40:16.264767 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264574    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="122898b2-56eb-4c82-9036-ddf9f6a246e4" containerName="payload-pre-processing"
Sep 16 21:40:16.264767 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264581    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a4fe71-7147-4091-831e-adc9a450c576" containerName="payload-processing"
Sep 16 21:40:16.264767 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264587    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d0a6186-19c9-4c74-8082-999432b88093" containerName="payload-pre-processing"
Sep 16 21:40:16.264767 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.264592    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="022a7697-6c58-4c3a-9e9e-b732fb7e1f6d" containerName="main"
Sep 16 21:40:16.269001 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.268981    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.271019 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.271002    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"e2e-conflict-b-f2f38651-kserve-self-signed-certs\""
Sep 16 21:40:16.275172 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.275153    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"]
Sep 16 21:40:16.417763 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.417724    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-model-cache\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.417763 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.417767    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-tmp-dir\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.417977 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.417814    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-home\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.417977 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.417855    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-dshm\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.417977 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.417921    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e804e81b-56a5-4454-99b3-2abd661e0f28-tls-certs\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.417977 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.417951    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6djx4\" (UniqueName: \"kubernetes.io/projected/e804e81b-56a5-4454-99b3-2abd661e0f28-kube-api-access-6djx4\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.519023 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.518940    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-model-cache\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.519023 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.518977    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-tmp-dir\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.519023 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.519024    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-home\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.519310 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.519056    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-dshm\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.519310 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.519094    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e804e81b-56a5-4454-99b3-2abd661e0f28-tls-certs\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.519310 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.519122    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6djx4\" (UniqueName: \"kubernetes.io/projected/e804e81b-56a5-4454-99b3-2abd661e0f28-kube-api-access-6djx4\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.519516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.519362    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-model-cache\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.519516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.519425    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-home\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.519516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.519447    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-tmp-dir\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.521324 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.521301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-dshm\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.521563 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.521539    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e804e81b-56a5-4454-99b3-2abd661e0f28-tls-certs\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.530646 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.530617    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6djx4\" (UniqueName: \"kubernetes.io/projected/e804e81b-56a5-4454-99b3-2abd661e0f28-kube-api-access-6djx4\") pod \"e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") " pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.579337 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.579303    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:16.854996 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:16.854964    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"]
Sep 16 21:40:16.857682 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:40:16.857656    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode804e81b_56a5_4454_99b3_2abd661e0f28.slice/crio-ff0abf2944f828cd8dbb483ddcde7d97e7c59f41280f1a9ea8997bcf8d261c2f WatchSource:0}: Error finding container ff0abf2944f828cd8dbb483ddcde7d97e7c59f41280f1a9ea8997bcf8d261c2f: Status 404 returned error can't find the container with id ff0abf2944f828cd8dbb483ddcde7d97e7c59f41280f1a9ea8997bcf8d261c2f
Sep 16 21:40:17.229769 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:17.229735    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm" event={"ID":"e804e81b-56a5-4454-99b3-2abd661e0f28","Type":"ContainerStarted","Data":"1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f"}
Sep 16 21:40:17.229917 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:17.229777    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm" event={"ID":"e804e81b-56a5-4454-99b3-2abd661e0f28","Type":"ContainerStarted","Data":"ff0abf2944f828cd8dbb483ddcde7d97e7c59f41280f1a9ea8997bcf8d261c2f"}
Sep 16 21:40:17.269956 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:17.269908    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm" podStartSLOduration=1.269893868 podStartE2EDuration="1.269893868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:17.2683585 +0000 UTC m=+1056.654041585" watchObservedRunningTime="2026-09-16 21:40:17.269893868 +0000 UTC m=+1056.655576951"
Sep 16 21:40:18.639057 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.639015    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"]
Sep 16 21:40:18.642907 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.642890    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:40:18.655711 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.655688    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-365ea1-dockercfg-g79xk\""
Sep 16 21:40:18.655810 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.655737    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-365ea1\""
Sep 16 21:40:18.666100 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.666075    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"]
Sep 16 21:40:18.681146 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.681123    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"]
Sep 16 21:40:18.684477 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.684460    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:40:18.697371 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.697351    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"]
Sep 16 21:40:18.740322 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.740292    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx68t\" (UniqueName: \"kubernetes.io/projected/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-kube-api-access-nx68t\") pod \"payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt\" (UID: \"466e6ec5-8ed5-47d4-a566-a009a0fadc4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:40:18.740501 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.740347    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt\" (UID: \"466e6ec5-8ed5-47d4-a566-a009a0fadc4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:40:18.841701 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.841665    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt\" (UID: \"466e6ec5-8ed5-47d4-a566-a009a0fadc4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:40:18.841854 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.841774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx68t\" (UniqueName: \"kubernetes.io/projected/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-kube-api-access-nx68t\") pod \"payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt\" (UID: \"466e6ec5-8ed5-47d4-a566-a009a0fadc4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:40:18.841854 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.841813    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8\" (UID: \"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:40:18.841854 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.841843    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfjvh\" (UniqueName: \"kubernetes.io/projected/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-kube-api-access-vfjvh\") pod \"payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8\" (UID: \"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:40:18.842365 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.842344    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt\" (UID: \"466e6ec5-8ed5-47d4-a566-a009a0fadc4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:40:18.850292 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.850261    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx68t\" (UniqueName: \"kubernetes.io/projected/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-kube-api-access-nx68t\") pod \"payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt\" (UID: \"466e6ec5-8ed5-47d4-a566-a009a0fadc4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:40:18.943169 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.943102    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8\" (UID: \"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:40:18.943169 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.943140    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfjvh\" (UniqueName: \"kubernetes.io/projected/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-kube-api-access-vfjvh\") pod \"payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8\" (UID: \"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:40:18.943800 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.943781    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8\" (UID: \"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:40:18.951430 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.951377    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfjvh\" (UniqueName: \"kubernetes.io/projected/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-kube-api-access-vfjvh\") pod \"payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8\" (UID: \"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:40:18.954055 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.954027    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:40:18.994734 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:18.994522    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:40:19.245739 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:19.245712    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"]
Sep 16 21:40:19.247056 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:40:19.247026    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466e6ec5_8ed5_47d4_a566_a009a0fadc4c.slice/crio-132d600a4403fc52bafec5acc69cb8ffac2e302c1f5747ed411462d9206be349 WatchSource:0}: Error finding container 132d600a4403fc52bafec5acc69cb8ffac2e302c1f5747ed411462d9206be349: Status 404 returned error can't find the container with id 132d600a4403fc52bafec5acc69cb8ffac2e302c1f5747ed411462d9206be349
Sep 16 21:40:19.290716 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:19.290692    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"]
Sep 16 21:40:19.291574 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:40:19.291547    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6cfca04_4a9c_4aa3_8d8e_0a35bd53405c.slice/crio-4fa76b00b23262b320a07c618af40efb5a48c9756a79bd9928ee27609a6b0847 WatchSource:0}: Error finding container 4fa76b00b23262b320a07c618af40efb5a48c9756a79bd9928ee27609a6b0847: Status 404 returned error can't find the container with id 4fa76b00b23262b320a07c618af40efb5a48c9756a79bd9928ee27609a6b0847
Sep 16 21:40:20.251580 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:20.251543    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt" event={"ID":"466e6ec5-8ed5-47d4-a566-a009a0fadc4c","Type":"ContainerStarted","Data":"ad5668f20409adc752484e13031dedf5996ce8ca66bfdd9786112082ddef404d"}
Sep 16 21:40:20.252002 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:20.251614    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:40:20.252002 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:20.251642    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt" event={"ID":"466e6ec5-8ed5-47d4-a566-a009a0fadc4c","Type":"ContainerStarted","Data":"132d600a4403fc52bafec5acc69cb8ffac2e302c1f5747ed411462d9206be349"}
Sep 16 21:40:20.253224 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:20.253194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8" event={"ID":"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c","Type":"ContainerStarted","Data":"1a5494ceff31c1a54dabe64ec1d575ed056e13e2516f8403266922b4f83d9d82"}
Sep 16 21:40:20.253345 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:20.253245    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8" event={"ID":"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c","Type":"ContainerStarted","Data":"4fa76b00b23262b320a07c618af40efb5a48c9756a79bd9928ee27609a6b0847"}
Sep 16 21:40:20.253345 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:20.253272    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:40:20.269803 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:20.269761    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt" podStartSLOduration=2.269746816 podStartE2EDuration="2.269746816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:20.268708033 +0000 UTC m=+1059.654391134" watchObservedRunningTime="2026-09-16 21:40:20.269746816 +0000 UTC m=+1059.655429899"
Sep 16 21:40:20.290224 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:20.290183    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8" podStartSLOduration=2.290169298 podStartE2EDuration="2.290169298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:20.28853322 +0000 UTC m=+1059.674216325" watchObservedRunningTime="2026-09-16 21:40:20.290169298 +0000 UTC m=+1059.675852393"
Sep 16 21:40:25.475889 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:25.475856    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:40:26.580165 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:26.580134    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:26.580571 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:26.580180    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:26.593905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:26.593881    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:27.308898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:27.308866    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:40:29.852209 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.852170    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"]
Sep 16 21:40:29.857314 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.857287    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:40:29.859853 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.859805    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a7df81\""
Sep 16 21:40:29.859956 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.859811    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a7df81-dockercfg-2khds\""
Sep 16 21:40:29.868535 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.868512    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"]
Sep 16 21:40:29.877202 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.877181    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"]
Sep 16 21:40:29.890853 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.890830    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:40:29.894146 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.894121    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"]
Sep 16 21:40:29.938417 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.938353    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cace92e0-5248-4fbe-84b8-534efbf590d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s\" (UID: \"cace92e0-5248-4fbe-84b8-534efbf590d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:40:29.938554 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.938438    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f00088-7730-4288-b556-fe91c9436ee9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k\" (UID: \"e6f00088-7730-4288-b556-fe91c9436ee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:40:29.938700 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.938678    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjt8d\" (UniqueName: \"kubernetes.io/projected/cace92e0-5248-4fbe-84b8-534efbf590d8-kube-api-access-cjt8d\") pod \"payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s\" (UID: \"cace92e0-5248-4fbe-84b8-534efbf590d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:40:29.939119 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:29.939072    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zdp\" (UniqueName: \"kubernetes.io/projected/e6f00088-7730-4288-b556-fe91c9436ee9-kube-api-access-z8zdp\") pod \"payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k\" (UID: \"e6f00088-7730-4288-b556-fe91c9436ee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:40:30.040220 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.040193    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f00088-7730-4288-b556-fe91c9436ee9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k\" (UID: \"e6f00088-7730-4288-b556-fe91c9436ee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:40:30.040329 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.040279    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjt8d\" (UniqueName: \"kubernetes.io/projected/cace92e0-5248-4fbe-84b8-534efbf590d8-kube-api-access-cjt8d\") pod \"payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s\" (UID: \"cace92e0-5248-4fbe-84b8-534efbf590d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:40:30.040329 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.040304    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zdp\" (UniqueName: \"kubernetes.io/projected/e6f00088-7730-4288-b556-fe91c9436ee9-kube-api-access-z8zdp\") pod \"payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k\" (UID: \"e6f00088-7730-4288-b556-fe91c9436ee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:40:30.040490 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.040339    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cace92e0-5248-4fbe-84b8-534efbf590d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s\" (UID: \"cace92e0-5248-4fbe-84b8-534efbf590d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:40:30.040901 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.040881    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f00088-7730-4288-b556-fe91c9436ee9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k\" (UID: \"e6f00088-7730-4288-b556-fe91c9436ee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:40:30.041020 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.041002    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cace92e0-5248-4fbe-84b8-534efbf590d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s\" (UID: \"cace92e0-5248-4fbe-84b8-534efbf590d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:40:30.052412 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.050010    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjt8d\" (UniqueName: \"kubernetes.io/projected/cace92e0-5248-4fbe-84b8-534efbf590d8-kube-api-access-cjt8d\") pod \"payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s\" (UID: \"cace92e0-5248-4fbe-84b8-534efbf590d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:40:30.055376 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.055351    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zdp\" (UniqueName: \"kubernetes.io/projected/e6f00088-7730-4288-b556-fe91c9436ee9-kube-api-access-z8zdp\") pod \"payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k\" (UID: \"e6f00088-7730-4288-b556-fe91c9436ee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:40:30.169232 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.169163    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:40:30.203040 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.203009    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:40:30.445484 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.445454    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"]
Sep 16 21:40:30.447974 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:40:30.447949    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcace92e0_5248_4fbe_84b8_534efbf590d8.slice/crio-174ce9bdfc53e16c08bf9853a2761f890a025eb84632ea26dfec07f50f5106f6 WatchSource:0}: Error finding container 174ce9bdfc53e16c08bf9853a2761f890a025eb84632ea26dfec07f50f5106f6: Status 404 returned error can't find the container with id 174ce9bdfc53e16c08bf9853a2761f890a025eb84632ea26dfec07f50f5106f6
Sep 16 21:40:30.498190 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:30.498166    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"]
Sep 16 21:40:30.499553 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:40:30.499529    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f00088_7730_4288_b556_fe91c9436ee9.slice/crio-48ef44156ca711546f15e8394e521a35e0f9073b2eb1bba941b9be4229d422a1 WatchSource:0}: Error finding container 48ef44156ca711546f15e8394e521a35e0f9073b2eb1bba941b9be4229d422a1: Status 404 returned error can't find the container with id 48ef44156ca711546f15e8394e521a35e0f9073b2eb1bba941b9be4229d422a1
Sep 16 21:40:31.262446 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:31.262418    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:40:31.262888 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:31.262485    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:40:31.330807 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:31.330770    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s" event={"ID":"cace92e0-5248-4fbe-84b8-534efbf590d8","Type":"ContainerStarted","Data":"5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293"}
Sep 16 21:40:31.330981 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:31.330830    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s" event={"ID":"cace92e0-5248-4fbe-84b8-534efbf590d8","Type":"ContainerStarted","Data":"174ce9bdfc53e16c08bf9853a2761f890a025eb84632ea26dfec07f50f5106f6"}
Sep 16 21:40:31.330981 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:31.330861    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:40:31.332883 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:31.332854    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k" event={"ID":"e6f00088-7730-4288-b556-fe91c9436ee9","Type":"ContainerStarted","Data":"111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59"}
Sep 16 21:40:31.333035 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:31.332908    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:40:31.333035 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:31.332933    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k" event={"ID":"e6f00088-7730-4288-b556-fe91c9436ee9","Type":"ContainerStarted","Data":"48ef44156ca711546f15e8394e521a35e0f9073b2eb1bba941b9be4229d422a1"}
Sep 16 21:40:31.349794 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:31.349747    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s" podStartSLOduration=2.3497269 podStartE2EDuration="2.3497269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:31.349591356 +0000 UTC m=+1070.735274466" watchObservedRunningTime="2026-09-16 21:40:31.3497269 +0000 UTC m=+1070.735409989"
Sep 16 21:40:31.368633 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:31.368587    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k" podStartSLOduration=2.368572176 podStartE2EDuration="2.368572176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:31.366014811 +0000 UTC m=+1070.751697912" watchObservedRunningTime="2026-09-16 21:40:31.368572176 +0000 UTC m=+1070.754255289"
Sep 16 21:40:42.342387 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:42.342356    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:40:42.342776 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:42.342530    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:40:53.880298 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:53.880260    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:40:54.748388 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.748352    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"]
Sep 16 21:40:54.752126 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.752102    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:40:54.754225 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.754199    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5ee44a89-dockercfg-g7nnb\""
Sep 16 21:40:54.754225 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.754217    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5ee44a89\""
Sep 16 21:40:54.762575 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.762555    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"]
Sep 16 21:40:54.784341 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.784316    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"]
Sep 16 21:40:54.788039 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.788023    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:40:54.799159 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.799129    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"]
Sep 16 21:40:54.865266 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.865232    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvvc\" (UniqueName: \"kubernetes.io/projected/90a261f6-e4b9-4389-8054-58e7eeb2f15c-kube-api-access-qnvvc\") pod \"payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k\" (UID: \"90a261f6-e4b9-4389-8054-58e7eeb2f15c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:40:54.865474 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.865358    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a261f6-e4b9-4389-8054-58e7eeb2f15c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k\" (UID: \"90a261f6-e4b9-4389-8054-58e7eeb2f15c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:40:54.865474 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.865391    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e16f7f9-f427-433a-a505-d90ab674fd52-plugins-config-volume\") pod \"payload-processing-e2e-mt-5ee44a89-8577659c69-r986q\" (UID: \"3e16f7f9-f427-433a-a505-d90ab674fd52\") " pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:40:54.865474 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.865458    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fszzt\" (UniqueName: \"kubernetes.io/projected/3e16f7f9-f427-433a-a505-d90ab674fd52-kube-api-access-fszzt\") pod \"payload-processing-e2e-mt-5ee44a89-8577659c69-r986q\" (UID: \"3e16f7f9-f427-433a-a505-d90ab674fd52\") " pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:40:54.966751 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.966717    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fszzt\" (UniqueName: \"kubernetes.io/projected/3e16f7f9-f427-433a-a505-d90ab674fd52-kube-api-access-fszzt\") pod \"payload-processing-e2e-mt-5ee44a89-8577659c69-r986q\" (UID: \"3e16f7f9-f427-433a-a505-d90ab674fd52\") " pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:40:54.967196 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.966886    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvvc\" (UniqueName: \"kubernetes.io/projected/90a261f6-e4b9-4389-8054-58e7eeb2f15c-kube-api-access-qnvvc\") pod \"payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k\" (UID: \"90a261f6-e4b9-4389-8054-58e7eeb2f15c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:40:54.967196 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.966976    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a261f6-e4b9-4389-8054-58e7eeb2f15c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k\" (UID: \"90a261f6-e4b9-4389-8054-58e7eeb2f15c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:40:54.967196 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.966998    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e16f7f9-f427-433a-a505-d90ab674fd52-plugins-config-volume\") pod \"payload-processing-e2e-mt-5ee44a89-8577659c69-r986q\" (UID: \"3e16f7f9-f427-433a-a505-d90ab674fd52\") " pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:40:54.967956 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.967935    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a261f6-e4b9-4389-8054-58e7eeb2f15c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k\" (UID: \"90a261f6-e4b9-4389-8054-58e7eeb2f15c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:40:54.968008 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.967935    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e16f7f9-f427-433a-a505-d90ab674fd52-plugins-config-volume\") pod \"payload-processing-e2e-mt-5ee44a89-8577659c69-r986q\" (UID: \"3e16f7f9-f427-433a-a505-d90ab674fd52\") " pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:40:54.974230 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.974199    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvvc\" (UniqueName: \"kubernetes.io/projected/90a261f6-e4b9-4389-8054-58e7eeb2f15c-kube-api-access-qnvvc\") pod \"payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k\" (UID: \"90a261f6-e4b9-4389-8054-58e7eeb2f15c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:40:54.974363 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:54.974340    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fszzt\" (UniqueName: \"kubernetes.io/projected/3e16f7f9-f427-433a-a505-d90ab674fd52-kube-api-access-fszzt\") pod \"payload-processing-e2e-mt-5ee44a89-8577659c69-r986q\" (UID: \"3e16f7f9-f427-433a-a505-d90ab674fd52\") " pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:40:55.064630 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.064568    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:40:55.098713 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.098682    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:40:55.343890 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.343861    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"]
Sep 16 21:40:55.344955 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:40:55.344929    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a261f6_e4b9_4389_8054_58e7eeb2f15c.slice/crio-089dacbecc5b76d7435fdfe71aeb16183bf559fc8048852ecbb756339e97847f WatchSource:0}: Error finding container 089dacbecc5b76d7435fdfe71aeb16183bf559fc8048852ecbb756339e97847f: Status 404 returned error can't find the container with id 089dacbecc5b76d7435fdfe71aeb16183bf559fc8048852ecbb756339e97847f
Sep 16 21:40:55.389655 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.389629    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"]
Sep 16 21:40:55.391514 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:40:55.391489    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e16f7f9_f427_433a_a505_d90ab674fd52.slice/crio-1e4610bc89c5cab0bc14b56f4a61be322a6dc6a2a742eac17bec7341d4633c17 WatchSource:0}: Error finding container 1e4610bc89c5cab0bc14b56f4a61be322a6dc6a2a742eac17bec7341d4633c17: Status 404 returned error can't find the container with id 1e4610bc89c5cab0bc14b56f4a61be322a6dc6a2a742eac17bec7341d4633c17
Sep 16 21:40:55.498668 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.498602    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k" event={"ID":"90a261f6-e4b9-4389-8054-58e7eeb2f15c","Type":"ContainerStarted","Data":"970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f"}
Sep 16 21:40:55.498668 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.498668    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k" event={"ID":"90a261f6-e4b9-4389-8054-58e7eeb2f15c","Type":"ContainerStarted","Data":"089dacbecc5b76d7435fdfe71aeb16183bf559fc8048852ecbb756339e97847f"}
Sep 16 21:40:55.498894 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.498774    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:40:55.500177 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.500147    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q" event={"ID":"3e16f7f9-f427-433a-a505-d90ab674fd52","Type":"ContainerStarted","Data":"d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e"}
Sep 16 21:40:55.500286 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.500201    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:40:55.500286 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.500220    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q" event={"ID":"3e16f7f9-f427-433a-a505-d90ab674fd52","Type":"ContainerStarted","Data":"1e4610bc89c5cab0bc14b56f4a61be322a6dc6a2a742eac17bec7341d4633c17"}
Sep 16 21:40:55.518936 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.518881    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k" podStartSLOduration=1.518861218 podStartE2EDuration="1.518861218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:55.514533943 +0000 UTC m=+1094.900217045" watchObservedRunningTime="2026-09-16 21:40:55.518861218 +0000 UTC m=+1094.904544303"
Sep 16 21:40:55.532896 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:40:55.532850    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q" podStartSLOduration=1.5328342080000001 podStartE2EDuration="1.532834208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:55.531257266 +0000 UTC m=+1094.916940369" watchObservedRunningTime="2026-09-16 21:40:55.532834208 +0000 UTC m=+1094.918517293"
Sep 16 21:41:01.556607 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.556568    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"]
Sep 16 21:41:01.560486 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.560469    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:41:01.563041 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.563016    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-1b68db-dockercfg-s68cp\""
Sep 16 21:41:01.563143 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.563023    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-1b68db\""
Sep 16 21:41:01.574193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.574171    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"]
Sep 16 21:41:01.734834 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.734796    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdhj\" (UniqueName: \"kubernetes.io/projected/f2108207-6b4e-4e1a-87b9-de4e665af470-kube-api-access-zgdhj\") pod \"payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6\" (UID: \"f2108207-6b4e-4e1a-87b9-de4e665af470\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:41:01.735004 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.734927    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2108207-6b4e-4e1a-87b9-de4e665af470-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6\" (UID: \"f2108207-6b4e-4e1a-87b9-de4e665af470\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:41:01.835849 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.835765    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdhj\" (UniqueName: \"kubernetes.io/projected/f2108207-6b4e-4e1a-87b9-de4e665af470-kube-api-access-zgdhj\") pod \"payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6\" (UID: \"f2108207-6b4e-4e1a-87b9-de4e665af470\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:41:01.836020 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.835858    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2108207-6b4e-4e1a-87b9-de4e665af470-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6\" (UID: \"f2108207-6b4e-4e1a-87b9-de4e665af470\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:41:01.836522 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.836502    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2108207-6b4e-4e1a-87b9-de4e665af470-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6\" (UID: \"f2108207-6b4e-4e1a-87b9-de4e665af470\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:41:01.844118 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.844091    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdhj\" (UniqueName: \"kubernetes.io/projected/f2108207-6b4e-4e1a-87b9-de4e665af470-kube-api-access-zgdhj\") pod \"payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6\" (UID: \"f2108207-6b4e-4e1a-87b9-de4e665af470\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:41:01.872940 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:01.872915    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:41:02.153111 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:02.153083    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"]
Sep 16 21:41:02.154231 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:41:02.154208    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2108207_6b4e_4e1a_87b9_de4e665af470.slice/crio-70f97e01e7e2fd52939d462de1bca5a9c38b731635b51f93d53baf5086be3bca WatchSource:0}: Error finding container 70f97e01e7e2fd52939d462de1bca5a9c38b731635b51f93d53baf5086be3bca: Status 404 returned error can't find the container with id 70f97e01e7e2fd52939d462de1bca5a9c38b731635b51f93d53baf5086be3bca
Sep 16 21:41:02.547090 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:02.547052    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6" event={"ID":"f2108207-6b4e-4e1a-87b9-de4e665af470","Type":"ContainerStarted","Data":"4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273"}
Sep 16 21:41:02.547250 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:02.547114    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6" event={"ID":"f2108207-6b4e-4e1a-87b9-de4e665af470","Type":"ContainerStarted","Data":"70f97e01e7e2fd52939d462de1bca5a9c38b731635b51f93d53baf5086be3bca"}
Sep 16 21:41:02.547250 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:02.547177    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:41:02.562873 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:02.562833    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6" podStartSLOduration=1.5628202839999998 podStartE2EDuration="1.562820284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:41:02.561512601 +0000 UTC m=+1101.947195696" watchObservedRunningTime="2026-09-16 21:41:02.562820284 +0000 UTC m=+1101.948503367"
Sep 16 21:41:06.508090 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:06.508054    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:41:06.508492 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:06.508149    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:41:13.556840 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:13.556810    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:41:19.109440 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:19.109411    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:41:51.985147 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:41:51.985111    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:42:07.959717 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:07.959635    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"]
Sep 16 21:42:22.979939 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:22.979896    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm" podUID="e804e81b-56a5-4454-99b3-2abd661e0f28" containerName="main" containerID="cri-o://1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f" gracePeriod=45
Sep 16 21:42:23.237247 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.237184    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:42:23.366408 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.366376    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-dshm\") pod \"e804e81b-56a5-4454-99b3-2abd661e0f28\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") "
Sep 16 21:42:23.366563 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.366442    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-tmp-dir\") pod \"e804e81b-56a5-4454-99b3-2abd661e0f28\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") "
Sep 16 21:42:23.366563 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.366482    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e804e81b-56a5-4454-99b3-2abd661e0f28-tls-certs\") pod \"e804e81b-56a5-4454-99b3-2abd661e0f28\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") "
Sep 16 21:42:23.366563 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.366514    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-home\") pod \"e804e81b-56a5-4454-99b3-2abd661e0f28\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") "
Sep 16 21:42:23.366737 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.366582    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6djx4\" (UniqueName: \"kubernetes.io/projected/e804e81b-56a5-4454-99b3-2abd661e0f28-kube-api-access-6djx4\") pod \"e804e81b-56a5-4454-99b3-2abd661e0f28\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") "
Sep 16 21:42:23.366737 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.366615    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-model-cache\") pod \"e804e81b-56a5-4454-99b3-2abd661e0f28\" (UID: \"e804e81b-56a5-4454-99b3-2abd661e0f28\") "
Sep 16 21:42:23.366737 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.366717    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e804e81b-56a5-4454-99b3-2abd661e0f28" (UID: "e804e81b-56a5-4454-99b3-2abd661e0f28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:42:23.366888 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.366789    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-home" (OuterVolumeSpecName: "home") pod "e804e81b-56a5-4454-99b3-2abd661e0f28" (UID: "e804e81b-56a5-4454-99b3-2abd661e0f28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:42:23.366958 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.366931    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-model-cache" (OuterVolumeSpecName: "model-cache") pod "e804e81b-56a5-4454-99b3-2abd661e0f28" (UID: "e804e81b-56a5-4454-99b3-2abd661e0f28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:42:23.367017 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.367007    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-tmp-dir\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:23.367073 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.367021    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-home\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:23.367073 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.367035    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-model-cache\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:23.368773 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.368748    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e804e81b-56a5-4454-99b3-2abd661e0f28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e804e81b-56a5-4454-99b3-2abd661e0f28" (UID: "e804e81b-56a5-4454-99b3-2abd661e0f28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:42:23.368906 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.368887    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-dshm" (OuterVolumeSpecName: "dshm") pod "e804e81b-56a5-4454-99b3-2abd661e0f28" (UID: "e804e81b-56a5-4454-99b3-2abd661e0f28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:42:23.368979 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.368962    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e804e81b-56a5-4454-99b3-2abd661e0f28-kube-api-access-6djx4" (OuterVolumeSpecName: "kube-api-access-6djx4") pod "e804e81b-56a5-4454-99b3-2abd661e0f28" (UID: "e804e81b-56a5-4454-99b3-2abd661e0f28"). InnerVolumeSpecName "kube-api-access-6djx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:42:23.467559 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.467528    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e804e81b-56a5-4454-99b3-2abd661e0f28-dshm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:23.467559 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.467554    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e804e81b-56a5-4454-99b3-2abd661e0f28-tls-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:23.467559 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:23.467564    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6djx4\" (UniqueName: \"kubernetes.io/projected/e804e81b-56a5-4454-99b3-2abd661e0f28-kube-api-access-6djx4\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:24.069229 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:24.069189    2567 generic.go:356] "Generic (PLEG): container finished" podID="e804e81b-56a5-4454-99b3-2abd661e0f28" containerID="1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f" exitCode=0
Sep 16 21:42:24.069638 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:24.069236    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"
Sep 16 21:42:24.069638 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:24.069273    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm" event={"ID":"e804e81b-56a5-4454-99b3-2abd661e0f28","Type":"ContainerDied","Data":"1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f"}
Sep 16 21:42:24.069638 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:24.069326    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm" event={"ID":"e804e81b-56a5-4454-99b3-2abd661e0f28","Type":"ContainerDied","Data":"ff0abf2944f828cd8dbb483ddcde7d97e7c59f41280f1a9ea8997bcf8d261c2f"}
Sep 16 21:42:24.069638 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:24.069345    2567 scope.go:117] "RemoveContainer" containerID="1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f"
Sep 16 21:42:24.079519 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:24.079501    2567 scope.go:117] "RemoveContainer" containerID="1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f"
Sep 16 21:42:24.079804 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:42:24.079784    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f\": container with ID starting with 1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f not found: ID does not exist" containerID="1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f"
Sep 16 21:42:24.079881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:24.079813    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f"} err="failed to get container status \"1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f\": rpc error: code = NotFound desc = could not find container \"1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f\": container with ID starting with 1047a3461e96ba2685e5676ec383610a7a9db2b22c3d892f7843a025bccfca6f not found: ID does not exist"
Sep 16 21:42:24.089784 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:24.089762    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"]
Sep 16 21:42:24.098713 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:24.093222    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-b-f2f38651-kserve-6fdbb89fcd-k2jhm"]
Sep 16 21:42:25.109950 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:25.109913    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e804e81b-56a5-4454-99b3-2abd661e0f28" path="/var/lib/kubelet/pods/e804e81b-56a5-4454-99b3-2abd661e0f28/volumes"
Sep 16 21:42:27.083055 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:27.083011    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:42:36.765543 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:36.765507    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"]
Sep 16 21:42:41.261144 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:41.261113    2567 scope.go:117] "RemoveContainer" containerID="7e0a5719f1ec43bb162317f338c7469913b9ca89354b28fd55dd1a9010c1b6f5"
Sep 16 21:42:51.784854 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:51.784794    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww" podUID="d1fc0d4c-0289-41ff-a701-6e4a50681111" containerName="main" containerID="cri-o://4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa" gracePeriod=45
Sep 16 21:42:52.048685 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.048626    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:42:52.126700 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.126667    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-tmp-dir\") pod \"d1fc0d4c-0289-41ff-a701-6e4a50681111\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") "
Sep 16 21:42:52.126842 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.126770    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fc0d4c-0289-41ff-a701-6e4a50681111-tls-certs\") pod \"d1fc0d4c-0289-41ff-a701-6e4a50681111\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") "
Sep 16 21:42:52.126842 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.126796    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-home\") pod \"d1fc0d4c-0289-41ff-a701-6e4a50681111\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") "
Sep 16 21:42:52.126842 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.126838    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-dshm\") pod \"d1fc0d4c-0289-41ff-a701-6e4a50681111\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") "
Sep 16 21:42:52.127012 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.126936    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1fc0d4c-0289-41ff-a701-6e4a50681111" (UID: "d1fc0d4c-0289-41ff-a701-6e4a50681111"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:42:52.127012 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.126999    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-model-cache\") pod \"d1fc0d4c-0289-41ff-a701-6e4a50681111\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") "
Sep 16 21:42:52.127133 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.127052    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n874\" (UniqueName: \"kubernetes.io/projected/d1fc0d4c-0289-41ff-a701-6e4a50681111-kube-api-access-5n874\") pod \"d1fc0d4c-0289-41ff-a701-6e4a50681111\" (UID: \"d1fc0d4c-0289-41ff-a701-6e4a50681111\") "
Sep 16 21:42:52.127133 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.127054    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-home" (OuterVolumeSpecName: "home") pod "d1fc0d4c-0289-41ff-a701-6e4a50681111" (UID: "d1fc0d4c-0289-41ff-a701-6e4a50681111"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:42:52.127244 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.127204    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1fc0d4c-0289-41ff-a701-6e4a50681111" (UID: "d1fc0d4c-0289-41ff-a701-6e4a50681111"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:42:52.127790 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.127539    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-home\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:52.127790 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.127565    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-model-cache\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:52.127790 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.127582    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-tmp-dir\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:52.129123 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.129099    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fc0d4c-0289-41ff-a701-6e4a50681111-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1fc0d4c-0289-41ff-a701-6e4a50681111" (UID: "d1fc0d4c-0289-41ff-a701-6e4a50681111"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:42:52.129295 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.129279    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-dshm" (OuterVolumeSpecName: "dshm") pod "d1fc0d4c-0289-41ff-a701-6e4a50681111" (UID: "d1fc0d4c-0289-41ff-a701-6e4a50681111"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:42:52.129769 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.129742    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fc0d4c-0289-41ff-a701-6e4a50681111-kube-api-access-5n874" (OuterVolumeSpecName: "kube-api-access-5n874") pod "d1fc0d4c-0289-41ff-a701-6e4a50681111" (UID: "d1fc0d4c-0289-41ff-a701-6e4a50681111"). InnerVolumeSpecName "kube-api-access-5n874". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:42:52.228739 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.228709    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fc0d4c-0289-41ff-a701-6e4a50681111-tls-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:52.228739 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.228734    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fc0d4c-0289-41ff-a701-6e4a50681111-dshm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:52.228881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.228743    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n874\" (UniqueName: \"kubernetes.io/projected/d1fc0d4c-0289-41ff-a701-6e4a50681111-kube-api-access-5n874\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:42:52.251269 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.251239    2567 generic.go:356] "Generic (PLEG): container finished" podID="d1fc0d4c-0289-41ff-a701-6e4a50681111" containerID="4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa" exitCode=0
Sep 16 21:42:52.251374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.251266    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww" event={"ID":"d1fc0d4c-0289-41ff-a701-6e4a50681111","Type":"ContainerDied","Data":"4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa"}
Sep 16 21:42:52.251374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.251287    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"
Sep 16 21:42:52.251374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.251306    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww" event={"ID":"d1fc0d4c-0289-41ff-a701-6e4a50681111","Type":"ContainerDied","Data":"772a6644f329bb17b2a7a0a80517083799bbe36a25ead6eb181f6ef05445175d"}
Sep 16 21:42:52.251374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.251324    2567 scope.go:117] "RemoveContainer" containerID="4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa"
Sep 16 21:42:52.261869 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.261852    2567 scope.go:117] "RemoveContainer" containerID="4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa"
Sep 16 21:42:52.262113 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:42:52.262096    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa\": container with ID starting with 4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa not found: ID does not exist" containerID="4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa"
Sep 16 21:42:52.262179 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.262119    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa"} err="failed to get container status \"4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa\": rpc error: code = NotFound desc = could not find container \"4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa\": container with ID starting with 4af75541a284ce8fa86f07b9244985fa8c9a549830bbfa24d11c7439401070aa not found: ID does not exist"
Sep 16 21:42:52.272938 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.272917    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"]
Sep 16 21:42:52.277039 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:52.277018    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/e2e-conflict-a-f2f38651-kserve-7bc5b9f888-xnfww"]
Sep 16 21:42:53.109327 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:42:53.109287    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1fc0d4c-0289-41ff-a701-6e4a50681111" path="/var/lib/kubelet/pods/d1fc0d4c-0289-41ff-a701-6e4a50681111/volumes"
Sep 16 21:43:02.078875 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:02.078833    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:43:06.048005 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.047968    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"]
Sep 16 21:43:06.048463 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.048449    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e804e81b-56a5-4454-99b3-2abd661e0f28" containerName="main"
Sep 16 21:43:06.048463 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.048463    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e804e81b-56a5-4454-99b3-2abd661e0f28" containerName="main"
Sep 16 21:43:06.048553 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.048477    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1fc0d4c-0289-41ff-a701-6e4a50681111" containerName="main"
Sep 16 21:43:06.048553 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.048482    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1fc0d4c-0289-41ff-a701-6e4a50681111" containerName="main"
Sep 16 21:43:06.048617 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.048563    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1fc0d4c-0289-41ff-a701-6e4a50681111" containerName="main"
Sep 16 21:43:06.048617 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.048572    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e804e81b-56a5-4454-99b3-2abd661e0f28" containerName="main"
Sep 16 21:43:06.053053 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.053035    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"
Sep 16 21:43:06.055158 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.055133    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:43:06.055269 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.055136    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:43:06.060368 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.060339    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"]
Sep 16 21:43:06.155393 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.155349    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17db2fde-21ba-4387-a968-ab33b24d7e75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5\" (UID: \"17db2fde-21ba-4387-a968-ab33b24d7e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"
Sep 16 21:43:06.256145 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.256110    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17db2fde-21ba-4387-a968-ab33b24d7e75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5\" (UID: \"17db2fde-21ba-4387-a968-ab33b24d7e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"
Sep 16 21:43:06.256753 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.256727    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17db2fde-21ba-4387-a968-ab33b24d7e75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5\" (UID: \"17db2fde-21ba-4387-a968-ab33b24d7e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"
Sep 16 21:43:06.364897 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.364832    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"
Sep 16 21:43:06.636037 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:06.636002    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"]
Sep 16 21:43:06.637200 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:43:06.637163    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17db2fde_21ba_4387_a968_ab33b24d7e75.slice/crio-59e1176080dc4767ce47f8542ca3ae0ab49dfb7daef1d58da1c4ba66ef0c338f WatchSource:0}: Error finding container 59e1176080dc4767ce47f8542ca3ae0ab49dfb7daef1d58da1c4ba66ef0c338f: Status 404 returned error can't find the container with id 59e1176080dc4767ce47f8542ca3ae0ab49dfb7daef1d58da1c4ba66ef0c338f
Sep 16 21:43:07.347477 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:07.347445    2567 generic.go:356] "Generic (PLEG): container finished" podID="17db2fde-21ba-4387-a968-ab33b24d7e75" containerID="4bb81e7744749d60e2e90e372d4e0b9fddef26b0df79521bc11978df3ed37617" exitCode=0
Sep 16 21:43:07.347863 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:07.347530    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5" event={"ID":"17db2fde-21ba-4387-a968-ab33b24d7e75","Type":"ContainerDied","Data":"4bb81e7744749d60e2e90e372d4e0b9fddef26b0df79521bc11978df3ed37617"}
Sep 16 21:43:07.347863 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:07.347578    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5" event={"ID":"17db2fde-21ba-4387-a968-ab33b24d7e75","Type":"ContainerStarted","Data":"59e1176080dc4767ce47f8542ca3ae0ab49dfb7daef1d58da1c4ba66ef0c338f"}
Sep 16 21:43:08.493765 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.493739    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"
Sep 16 21:43:08.576567 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.576532    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17db2fde-21ba-4387-a968-ab33b24d7e75-maas-api-service-ca\") pod \"17db2fde-21ba-4387-a968-ab33b24d7e75\" (UID: \"17db2fde-21ba-4387-a968-ab33b24d7e75\") "
Sep 16 21:43:08.576852 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.576828    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17db2fde-21ba-4387-a968-ab33b24d7e75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17db2fde-21ba-4387-a968-ab33b24d7e75" (UID: "17db2fde-21ba-4387-a968-ab33b24d7e75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:43:08.678147 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.678067    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17db2fde-21ba-4387-a968-ab33b24d7e75-maas-api-service-ca\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:43:08.885096 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.885059    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"]
Sep 16 21:43:08.885627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.885612    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17db2fde-21ba-4387-a968-ab33b24d7e75" containerName="revoke-keys"
Sep 16 21:43:08.885627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.885628    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17db2fde-21ba-4387-a968-ab33b24d7e75" containerName="revoke-keys"
Sep 16 21:43:08.885727 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.885707    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="17db2fde-21ba-4387-a968-ab33b24d7e75" containerName="revoke-keys"
Sep 16 21:43:08.888859 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.888841    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:08.891406 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.891372    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"distinct-w5-e047d8-kserve-self-signed-certs\""
Sep 16 21:43:08.899627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.899607    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"]
Sep 16 21:43:08.981129 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.981095    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-home\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:08.981285 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.981138    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22mqj\" (UniqueName: \"kubernetes.io/projected/01414087-99a7-46bd-ae45-67ecb110c13c-kube-api-access-22mqj\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:08.981285 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.981178    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-dshm\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:08.981285 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.981251    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-tmp-dir\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:08.981433 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.981292    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01414087-99a7-46bd-ae45-67ecb110c13c-tls-certs\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:08.981433 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:08.981378    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-model-cache\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.082366 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.082325    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-home\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.082366 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.082368    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22mqj\" (UniqueName: \"kubernetes.io/projected/01414087-99a7-46bd-ae45-67ecb110c13c-kube-api-access-22mqj\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.082698 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.082447    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-dshm\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.082698 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.082507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-tmp-dir\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.082698 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.082551    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01414087-99a7-46bd-ae45-67ecb110c13c-tls-certs\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.082698 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.082596    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-model-cache\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.083063 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.083037    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-home\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.084050 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.083979    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-tmp-dir\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.084050 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.083984    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-model-cache\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.085061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.085041    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-dshm\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.085554 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.085537    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01414087-99a7-46bd-ae45-67ecb110c13c-tls-certs\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.096884 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.096859    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22mqj\" (UniqueName: \"kubernetes.io/projected/01414087-99a7-46bd-ae45-67ecb110c13c-kube-api-access-22mqj\") pod \"distinct-w5-e047d8-kserve-64fc84f546-s7t2h\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.198409 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.198362    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:09.360702 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.360663    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5" event={"ID":"17db2fde-21ba-4387-a968-ab33b24d7e75","Type":"ContainerDied","Data":"59e1176080dc4767ce47f8542ca3ae0ab49dfb7daef1d58da1c4ba66ef0c338f"}
Sep 16 21:43:09.360702 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.360690    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"
Sep 16 21:43:09.360909 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.360718    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e1176080dc4767ce47f8542ca3ae0ab49dfb7daef1d58da1c4ba66ef0c338f"
Sep 16 21:43:09.473869 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:09.473845    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"]
Sep 16 21:43:09.474609 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:43:09.474579    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01414087_99a7_46bd_ae45_67ecb110c13c.slice/crio-23d4d0cf6ce3b0d939f641da54d5b6f29c94882e6955250a715865f559ea632c WatchSource:0}: Error finding container 23d4d0cf6ce3b0d939f641da54d5b6f29c94882e6955250a715865f559ea632c: Status 404 returned error can't find the container with id 23d4d0cf6ce3b0d939f641da54d5b6f29c94882e6955250a715865f559ea632c
Sep 16 21:43:10.368635 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:10.368593    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h" event={"ID":"01414087-99a7-46bd-ae45-67ecb110c13c","Type":"ContainerStarted","Data":"4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393"}
Sep 16 21:43:10.368996 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:10.368643    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h" event={"ID":"01414087-99a7-46bd-ae45-67ecb110c13c","Type":"ContainerStarted","Data":"23d4d0cf6ce3b0d939f641da54d5b6f29c94882e6955250a715865f559ea632c"}
Sep 16 21:43:10.387469 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:10.387424    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h" podStartSLOduration=2.387407626 podStartE2EDuration="2.387407626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:43:10.384514782 +0000 UTC m=+1229.770197866" watchObservedRunningTime="2026-09-16 21:43:10.387407626 +0000 UTC m=+1229.773090702"
Sep 16 21:43:19.198649 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:19.198616    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:19.199107 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:19.198667    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:19.211049 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:19.211027    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:19.435021 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:19.434984    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:43:21.704891 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:21.704858    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"]
Sep 16 21:43:21.705347 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:21.705164    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q" podUID="3e16f7f9-f427-433a-a505-d90ab674fd52" containerName="payload-processing" containerID="cri-o://d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e" gracePeriod=30
Sep 16 21:43:21.714714 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:21.714685    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"]
Sep 16 21:43:21.715051 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:21.714910    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k" podUID="90a261f6-e4b9-4389-8054-58e7eeb2f15c" containerName="payload-pre-processing" containerID="cri-o://970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f" gracePeriod=30
Sep 16 21:43:21.894521 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:21.894374    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"]
Sep 16 21:43:21.897266 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:21.897238    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ee44a89-dh9x5"]
Sep 16 21:43:21.997930 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:21.997895    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:43:22.001492 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.001460    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:43:22.098544 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.098511    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a261f6-e4b9-4389-8054-58e7eeb2f15c-plugins-config-volume\") pod \"90a261f6-e4b9-4389-8054-58e7eeb2f15c\" (UID: \"90a261f6-e4b9-4389-8054-58e7eeb2f15c\") "
Sep 16 21:43:22.098729 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.098559    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnvvc\" (UniqueName: \"kubernetes.io/projected/90a261f6-e4b9-4389-8054-58e7eeb2f15c-kube-api-access-qnvvc\") pod \"90a261f6-e4b9-4389-8054-58e7eeb2f15c\" (UID: \"90a261f6-e4b9-4389-8054-58e7eeb2f15c\") "
Sep 16 21:43:22.098729 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.098704    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e16f7f9-f427-433a-a505-d90ab674fd52-plugins-config-volume\") pod \"3e16f7f9-f427-433a-a505-d90ab674fd52\" (UID: \"3e16f7f9-f427-433a-a505-d90ab674fd52\") "
Sep 16 21:43:22.098729 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.098727    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fszzt\" (UniqueName: \"kubernetes.io/projected/3e16f7f9-f427-433a-a505-d90ab674fd52-kube-api-access-fszzt\") pod \"3e16f7f9-f427-433a-a505-d90ab674fd52\" (UID: \"3e16f7f9-f427-433a-a505-d90ab674fd52\") "
Sep 16 21:43:22.098956 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.098929    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a261f6-e4b9-4389-8054-58e7eeb2f15c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90a261f6-e4b9-4389-8054-58e7eeb2f15c" (UID: "90a261f6-e4b9-4389-8054-58e7eeb2f15c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:43:22.099103 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.099076    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a261f6-e4b9-4389-8054-58e7eeb2f15c-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:43:22.099229 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.099130    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e16f7f9-f427-433a-a505-d90ab674fd52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e16f7f9-f427-433a-a505-d90ab674fd52" (UID: "3e16f7f9-f427-433a-a505-d90ab674fd52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:43:22.100811 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.100784    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a261f6-e4b9-4389-8054-58e7eeb2f15c-kube-api-access-qnvvc" (OuterVolumeSpecName: "kube-api-access-qnvvc") pod "90a261f6-e4b9-4389-8054-58e7eeb2f15c" (UID: "90a261f6-e4b9-4389-8054-58e7eeb2f15c"). InnerVolumeSpecName "kube-api-access-qnvvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:43:22.100898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.100793    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e16f7f9-f427-433a-a505-d90ab674fd52-kube-api-access-fszzt" (OuterVolumeSpecName: "kube-api-access-fszzt") pod "3e16f7f9-f427-433a-a505-d90ab674fd52" (UID: "3e16f7f9-f427-433a-a505-d90ab674fd52"). InnerVolumeSpecName "kube-api-access-fszzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:43:22.200452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.200421    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e16f7f9-f427-433a-a505-d90ab674fd52-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:43:22.200452 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.200450    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fszzt\" (UniqueName: \"kubernetes.io/projected/3e16f7f9-f427-433a-a505-d90ab674fd52-kube-api-access-fszzt\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:43:22.200653 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.200526    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnvvc\" (UniqueName: \"kubernetes.io/projected/90a261f6-e4b9-4389-8054-58e7eeb2f15c-kube-api-access-qnvvc\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:43:22.445515 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.445429    2567 generic.go:356] "Generic (PLEG): container finished" podID="90a261f6-e4b9-4389-8054-58e7eeb2f15c" containerID="970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f" exitCode=0
Sep 16 21:43:22.445515 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.445467    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k" event={"ID":"90a261f6-e4b9-4389-8054-58e7eeb2f15c","Type":"ContainerDied","Data":"970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f"}
Sep 16 21:43:22.445515 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.445516    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k" event={"ID":"90a261f6-e4b9-4389-8054-58e7eeb2f15c","Type":"ContainerDied","Data":"089dacbecc5b76d7435fdfe71aeb16183bf559fc8048852ecbb756339e97847f"}
Sep 16 21:43:22.445777 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.445450    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"
Sep 16 21:43:22.445777 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.445542    2567 scope.go:117] "RemoveContainer" containerID="970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f"
Sep 16 21:43:22.447182 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.447151    2567 generic.go:356] "Generic (PLEG): container finished" podID="3e16f7f9-f427-433a-a505-d90ab674fd52" containerID="d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e" exitCode=0
Sep 16 21:43:22.447308 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.447202    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"
Sep 16 21:43:22.447308 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.447205    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q" event={"ID":"3e16f7f9-f427-433a-a505-d90ab674fd52","Type":"ContainerDied","Data":"d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e"}
Sep 16 21:43:22.447308 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.447295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q" event={"ID":"3e16f7f9-f427-433a-a505-d90ab674fd52","Type":"ContainerDied","Data":"1e4610bc89c5cab0bc14b56f4a61be322a6dc6a2a742eac17bec7341d4633c17"}
Sep 16 21:43:22.457652 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.457636    2567 scope.go:117] "RemoveContainer" containerID="970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f"
Sep 16 21:43:22.457969 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:43:22.457952    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f\": container with ID starting with 970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f not found: ID does not exist" containerID="970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f"
Sep 16 21:43:22.458054 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.457981    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f"} err="failed to get container status \"970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f\": rpc error: code = NotFound desc = could not find container \"970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f\": container with ID starting with 970a62e34a4b14e4a9580d5a50f09713a179a9465dc3a106d7465ee8de71c83f not found: ID does not exist"
Sep 16 21:43:22.458054 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.458003    2567 scope.go:117] "RemoveContainer" containerID="d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e"
Sep 16 21:43:22.466387 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.466369    2567 scope.go:117] "RemoveContainer" containerID="d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e"
Sep 16 21:43:22.466692 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:43:22.466673    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e\": container with ID starting with d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e not found: ID does not exist" containerID="d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e"
Sep 16 21:43:22.466753 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.466701    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e"} err="failed to get container status \"d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e\": rpc error: code = NotFound desc = could not find container \"d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e\": container with ID starting with d9caab71693bce6b4a26810c967a503ddefd5c3831bb30a9cef843c99e1fe52e not found: ID does not exist"
Sep 16 21:43:22.470646 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.470623    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"]
Sep 16 21:43:22.474507 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.474487    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ee44a89-5f84f9dbf4-hn84k"]
Sep 16 21:43:22.486817 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.486792    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"]
Sep 16 21:43:22.490405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:22.490375    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ee44a89-8577659c69-r986q"]
Sep 16 21:43:23.108911 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:23.108877    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17db2fde-21ba-4387-a968-ab33b24d7e75" path="/var/lib/kubelet/pods/17db2fde-21ba-4387-a968-ab33b24d7e75/volumes"
Sep 16 21:43:23.109329 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:23.109223    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e16f7f9-f427-433a-a505-d90ab674fd52" path="/var/lib/kubelet/pods/3e16f7f9-f427-433a-a505-d90ab674fd52/volumes"
Sep 16 21:43:23.109585 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:23.109570    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90a261f6-e4b9-4389-8054-58e7eeb2f15c" path="/var/lib/kubelet/pods/90a261f6-e4b9-4389-8054-58e7eeb2f15c/volumes"
Sep 16 21:43:25.379785 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.379751    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"]
Sep 16 21:43:25.380212 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.380197    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a261f6-e4b9-4389-8054-58e7eeb2f15c" containerName="payload-pre-processing"
Sep 16 21:43:25.380212 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.380213    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a261f6-e4b9-4389-8054-58e7eeb2f15c" containerName="payload-pre-processing"
Sep 16 21:43:25.380306 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.380229    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e16f7f9-f427-433a-a505-d90ab674fd52" containerName="payload-processing"
Sep 16 21:43:25.380306 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.380236    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e16f7f9-f427-433a-a505-d90ab674fd52" containerName="payload-processing"
Sep 16 21:43:25.380306 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.380303    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e16f7f9-f427-433a-a505-d90ab674fd52" containerName="payload-processing"
Sep 16 21:43:25.380434 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.380314    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="90a261f6-e4b9-4389-8054-58e7eeb2f15c" containerName="payload-pre-processing"
Sep 16 21:43:25.384880 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.384861    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.387112 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.387092    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"kube-root-ca.crt\""
Sep 16 21:43:25.387775 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.387757    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"openshift-service-ca.crt\""
Sep 16 21:43:25.387832 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.387802    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"default-dockercfg-xb2m5\""
Sep 16 21:43:25.387898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.387877    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"auth-test-model-1b68db-kserve-self-signed-certs\""
Sep 16 21:43:25.393086 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.393063    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"]
Sep 16 21:43:25.531048 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.531007    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tls-certs\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.531048 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.531047    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-dshm\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.531299 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.531117    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-model-cache\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.531299 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.531157    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drv95\" (UniqueName: \"kubernetes.io/projected/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-kube-api-access-drv95\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.531299 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.531184    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-home\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.531299 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.531211    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tmp-dir\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.631788 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.631701    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drv95\" (UniqueName: \"kubernetes.io/projected/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-kube-api-access-drv95\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.631788 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.631743    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-home\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.631788 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.631770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tmp-dir\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.632067 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.631837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tls-certs\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.632067 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.631879    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-dshm\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.632067 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.631931    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-model-cache\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.632234 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.632165    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-home\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.632292 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.632249    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tmp-dir\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.632349 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.632314    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-model-cache\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.634082 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.634057    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-dshm\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.634458 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.634434    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tls-certs\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.639853 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.639807    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drv95\" (UniqueName: \"kubernetes.io/projected/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-kube-api-access-drv95\") pod \"auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.695929 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.695898    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:25.969478 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:25.969451    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"]
Sep 16 21:43:25.970936 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:43:25.970904    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b6d29b2_8aec_4bf5_a69b_24fecc57a19f.slice/crio-9f729ef3b6ad36194861714eb1e53ef70169659ad217acf46292cdcf0a613c18 WatchSource:0}: Error finding container 9f729ef3b6ad36194861714eb1e53ef70169659ad217acf46292cdcf0a613c18: Status 404 returned error can't find the container with id 9f729ef3b6ad36194861714eb1e53ef70169659ad217acf46292cdcf0a613c18
Sep 16 21:43:26.482481 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:26.482445    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9" event={"ID":"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f","Type":"ContainerStarted","Data":"9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce"}
Sep 16 21:43:26.482481 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:26.482494    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9" event={"ID":"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f","Type":"ContainerStarted","Data":"9f729ef3b6ad36194861714eb1e53ef70169659ad217acf46292cdcf0a613c18"}
Sep 16 21:43:26.498480 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:26.498434    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9" podStartSLOduration=1.498419187 podStartE2EDuration="1.498419187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:43:26.496604201 +0000 UTC m=+1245.882287304" watchObservedRunningTime="2026-09-16 21:43:26.498419187 +0000 UTC m=+1245.884102268"
Sep 16 21:43:34.982952 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:34.982869    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:43:35.696783 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:35.696747    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:35.696783 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:35.696792    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:35.709672 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:35.709646    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:43:36.557413 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:43:36.557371    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:44:05.677785 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.677741    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"]
Sep 16 21:44:05.682978 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.682958    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.685043 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.685025    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"distinct-2-w5-e047d8-kserve-self-signed-certs\""
Sep 16 21:44:05.689121 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.689100    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"]
Sep 16 21:44:05.779175 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.779138    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-tmp-dir\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.779175 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.779176    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d150203a-6382-4b30-94e1-16131d9f64d5-tls-certs\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.779375 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.779284    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-home\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.779375 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.779309    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-dshm\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.779375 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.779346    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k5pc\" (UniqueName: \"kubernetes.io/projected/d150203a-6382-4b30-94e1-16131d9f64d5-kube-api-access-5k5pc\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.779495 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.779420    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-model-cache\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.880351 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.880312    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-home\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.880351 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.880353    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-dshm\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.880634 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.880467    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k5pc\" (UniqueName: \"kubernetes.io/projected/d150203a-6382-4b30-94e1-16131d9f64d5-kube-api-access-5k5pc\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.880634 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.880516    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-model-cache\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.880765 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.880632    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-tmp-dir\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.880765 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.880756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d150203a-6382-4b30-94e1-16131d9f64d5-tls-certs\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.880876 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.880824    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-home\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.880942 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.880930    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-model-cache\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.881297 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.881270    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-tmp-dir\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.882815 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.882793    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-dshm\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.883386 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.883364    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d150203a-6382-4b30-94e1-16131d9f64d5-tls-certs\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.888287 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.888267    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k5pc\" (UniqueName: \"kubernetes.io/projected/d150203a-6382-4b30-94e1-16131d9f64d5-kube-api-access-5k5pc\") pod \"distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") " pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:05.994258 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:05.994223    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:06.267207 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:06.267178    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"]
Sep 16 21:44:06.268619 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:44:06.268590    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd150203a_6382_4b30_94e1_16131d9f64d5.slice/crio-bf3bc96fb207564d7b21ebd5b10ec80e5424a45e76ce694543229e6bbd31c315 WatchSource:0}: Error finding container bf3bc96fb207564d7b21ebd5b10ec80e5424a45e76ce694543229e6bbd31c315: Status 404 returned error can't find the container with id bf3bc96fb207564d7b21ebd5b10ec80e5424a45e76ce694543229e6bbd31c315
Sep 16 21:44:06.746082 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:06.746048    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd" event={"ID":"d150203a-6382-4b30-94e1-16131d9f64d5","Type":"ContainerStarted","Data":"d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882"}
Sep 16 21:44:06.746082 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:06.746090    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd" event={"ID":"d150203a-6382-4b30-94e1-16131d9f64d5","Type":"ContainerStarted","Data":"bf3bc96fb207564d7b21ebd5b10ec80e5424a45e76ce694543229e6bbd31c315"}
Sep 16 21:44:06.763672 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:06.763624    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd" podStartSLOduration=1.7636098919999998 podStartE2EDuration="1.763609892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:44:06.761501101 +0000 UTC m=+1286.147184186" watchObservedRunningTime="2026-09-16 21:44:06.763609892 +0000 UTC m=+1286.149292976"
Sep 16 21:44:06.885184 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:06.885155    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:44:15.995377 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:15.995338    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:15.995774 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:15.995421    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:16.007708 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:16.007681    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:16.816159 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:16.816130    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:44:41.310757 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:41.310726    2567 scope.go:117] "RemoveContainer" containerID="febf613a011642565f366712023bc3235f0fc088c2822a78eb5678cd33cc4eec"
Sep 16 21:44:42.490797 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:44:42.490755    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:45:00.143507 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.143469    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp"]
Sep 16 21:45:00.148984 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.148957    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc"]
Sep 16 21:45:00.149149 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.149070    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp"
Sep 16 21:45:00.151670 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.151641    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:45:00.153010 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.152983    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc"
Sep 16 21:45:00.159914 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.158996    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp"]
Sep 16 21:45:00.162954 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.162825    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn"]
Sep 16 21:45:00.166655 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.166630    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp"
Sep 16 21:45:00.166926 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.166909    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc"]
Sep 16 21:45:00.167012 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.166995    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn"
Sep 16 21:45:00.177200 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.177123    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn"]
Sep 16 21:45:00.178925 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.178904    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc"
Sep 16 21:45:00.219915 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.219892    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn"
Sep 16 21:45:00.454288 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.454259    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp"]
Sep 16 21:45:00.455952 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:45:00.455927    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c80c73_257a_4747_93c9_86fbeb88d781.slice/crio-b1a8bccbe8f6a97c6f2d0d73bf25a96e24160c2a3896f15ff480b87747444203 WatchSource:0}: Error finding container b1a8bccbe8f6a97c6f2d0d73bf25a96e24160c2a3896f15ff480b87747444203: Status 404 returned error can't find the container with id b1a8bccbe8f6a97c6f2d0d73bf25a96e24160c2a3896f15ff480b87747444203
Sep 16 21:45:00.478888 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.478867    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc"]
Sep 16 21:45:00.491430 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:45:00.491381    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c672b65_67d7_4929_b31c_03ce160562df.slice/crio-0c281af7ba8b01170cc06de4d0fbef3faaf348d3029716cd2b6d2ccb4d7f908e WatchSource:0}: Error finding container 0c281af7ba8b01170cc06de4d0fbef3faaf348d3029716cd2b6d2ccb4d7f908e: Status 404 returned error can't find the container with id 0c281af7ba8b01170cc06de4d0fbef3faaf348d3029716cd2b6d2ccb4d7f908e
Sep 16 21:45:00.519251 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:00.519228    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn"]
Sep 16 21:45:00.527654 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:45:00.527631    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fd6533_27b6_4f40_b4c4_ebcfcf1984cb.slice/crio-1f028637bd3c1c6be77ab51263d9712f68226acf2258c732ecea763076f15d25 WatchSource:0}: Error finding container 1f028637bd3c1c6be77ab51263d9712f68226acf2258c732ecea763076f15d25: Status 404 returned error can't find the container with id 1f028637bd3c1c6be77ab51263d9712f68226acf2258c732ecea763076f15d25
Sep 16 21:45:01.089954 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:01.089865    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp" event={"ID":"c3c80c73-257a-4747-93c9-86fbeb88d781","Type":"ContainerStarted","Data":"1066ef53cf07d8ad3df3d23290b236100a40d6b238b94bf0199ecf3e9e3d9d1d"}
Sep 16 21:45:01.089954 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:01.089920    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp" event={"ID":"c3c80c73-257a-4747-93c9-86fbeb88d781","Type":"ContainerStarted","Data":"b1a8bccbe8f6a97c6f2d0d73bf25a96e24160c2a3896f15ff480b87747444203"}
Sep 16 21:45:01.091391 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:01.091366    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn" event={"ID":"50fd6533-27b6-4f40-b4c4-ebcfcf1984cb","Type":"ContainerStarted","Data":"44da05de4980dbe74b6fc32f3869e3b2b17819def997a751ecb2e94f9968fb64"}
Sep 16 21:45:01.091518 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:01.091437    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn" event={"ID":"50fd6533-27b6-4f40-b4c4-ebcfcf1984cb","Type":"ContainerStarted","Data":"1f028637bd3c1c6be77ab51263d9712f68226acf2258c732ecea763076f15d25"}
Sep 16 21:45:01.092935 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:01.092913    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc" event={"ID":"6c672b65-67d7-4929-b31c-03ce160562df","Type":"ContainerStarted","Data":"2ae0f022424901f4f449206d4f418924d1938ac2980201b2ab6c931a5c19ea66"}
Sep 16 21:45:01.093021 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:01.092945    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc" event={"ID":"6c672b65-67d7-4929-b31c-03ce160562df","Type":"ContainerStarted","Data":"0c281af7ba8b01170cc06de4d0fbef3faaf348d3029716cd2b6d2ccb4d7f908e"}
Sep 16 21:45:01.106373 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:01.106318    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp" podStartSLOduration=1.106300901 podStartE2EDuration="1.106300901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:01.103964985 +0000 UTC m=+1340.489648088" watchObservedRunningTime="2026-09-16 21:45:01.106300901 +0000 UTC m=+1340.491983985"
Sep 16 21:45:01.126564 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:01.126511    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc" podStartSLOduration=1.126491321 podStartE2EDuration="1.126491321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:01.123295627 +0000 UTC m=+1340.508978712" watchObservedRunningTime="2026-09-16 21:45:01.126491321 +0000 UTC m=+1340.512174406"
Sep 16 21:45:01.141349 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:01.141264    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn" podStartSLOduration=1.14124337 podStartE2EDuration="1.14124337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:01.136947756 +0000 UTC m=+1340.522630860" watchObservedRunningTime="2026-09-16 21:45:01.14124337 +0000 UTC m=+1340.526926455"
Sep 16 21:45:02.100584 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:02.100549    2567 generic.go:356] "Generic (PLEG): container finished" podID="c3c80c73-257a-4747-93c9-86fbeb88d781" containerID="1066ef53cf07d8ad3df3d23290b236100a40d6b238b94bf0199ecf3e9e3d9d1d" exitCode=0
Sep 16 21:45:02.100974 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:02.100636    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp" event={"ID":"c3c80c73-257a-4747-93c9-86fbeb88d781","Type":"ContainerDied","Data":"1066ef53cf07d8ad3df3d23290b236100a40d6b238b94bf0199ecf3e9e3d9d1d"}
Sep 16 21:45:02.102268 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:02.102243    2567 generic.go:356] "Generic (PLEG): container finished" podID="50fd6533-27b6-4f40-b4c4-ebcfcf1984cb" containerID="44da05de4980dbe74b6fc32f3869e3b2b17819def997a751ecb2e94f9968fb64" exitCode=0
Sep 16 21:45:02.102438 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:02.102307    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn" event={"ID":"50fd6533-27b6-4f40-b4c4-ebcfcf1984cb","Type":"ContainerDied","Data":"44da05de4980dbe74b6fc32f3869e3b2b17819def997a751ecb2e94f9968fb64"}
Sep 16 21:45:02.103913 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:02.103891    2567 generic.go:356] "Generic (PLEG): container finished" podID="6c672b65-67d7-4929-b31c-03ce160562df" containerID="2ae0f022424901f4f449206d4f418924d1938ac2980201b2ab6c931a5c19ea66" exitCode=0
Sep 16 21:45:02.103991 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:02.103971    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc" event={"ID":"6c672b65-67d7-4929-b31c-03ce160562df","Type":"ContainerDied","Data":"2ae0f022424901f4f449206d4f418924d1938ac2980201b2ab6c931a5c19ea66"}
Sep 16 21:45:03.274073 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:03.274047    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn"
Sep 16 21:45:03.320609 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:03.320581    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp"
Sep 16 21:45:03.323691 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:03.323670    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc"
Sep 16 21:45:04.121335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:04.121306    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp"
Sep 16 21:45:04.121512 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:04.121309    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp" event={"ID":"c3c80c73-257a-4747-93c9-86fbeb88d781","Type":"ContainerDied","Data":"b1a8bccbe8f6a97c6f2d0d73bf25a96e24160c2a3896f15ff480b87747444203"}
Sep 16 21:45:04.121512 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:04.121455    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1a8bccbe8f6a97c6f2d0d73bf25a96e24160c2a3896f15ff480b87747444203"
Sep 16 21:45:04.122881 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:04.122857    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn"
Sep 16 21:45:04.123017 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:04.122861    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn" event={"ID":"50fd6533-27b6-4f40-b4c4-ebcfcf1984cb","Type":"ContainerDied","Data":"1f028637bd3c1c6be77ab51263d9712f68226acf2258c732ecea763076f15d25"}
Sep 16 21:45:04.123017 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:04.122994    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f028637bd3c1c6be77ab51263d9712f68226acf2258c732ecea763076f15d25"
Sep 16 21:45:04.124568 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:04.124492    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc"
Sep 16 21:45:04.124568 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:04.124542    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc" event={"ID":"6c672b65-67d7-4929-b31c-03ce160562df","Type":"ContainerDied","Data":"0c281af7ba8b01170cc06de4d0fbef3faaf348d3029716cd2b6d2ccb4d7f908e"}
Sep 16 21:45:04.124721 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:04.124588    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c281af7ba8b01170cc06de4d0fbef3faaf348d3029716cd2b6d2ccb4d7f908e"
Sep 16 21:45:16.597129 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.597094    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"]
Sep 16 21:45:16.597663 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.597571    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c80c73-257a-4747-93c9-86fbeb88d781" containerName="cleanup"
Sep 16 21:45:16.597663 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.597583    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c80c73-257a-4747-93c9-86fbeb88d781" containerName="cleanup"
Sep 16 21:45:16.597663 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.597598    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c672b65-67d7-4929-b31c-03ce160562df" containerName="cleanup"
Sep 16 21:45:16.597663 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.597603    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c672b65-67d7-4929-b31c-03ce160562df" containerName="cleanup"
Sep 16 21:45:16.597663 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.597613    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fd6533-27b6-4f40-b4c4-ebcfcf1984cb" containerName="cleanup"
Sep 16 21:45:16.597663 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.597619    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fd6533-27b6-4f40-b4c4-ebcfcf1984cb" containerName="cleanup"
Sep 16 21:45:16.597876 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.597681    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c80c73-257a-4747-93c9-86fbeb88d781" containerName="cleanup"
Sep 16 21:45:16.597876 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.597687    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="50fd6533-27b6-4f40-b4c4-ebcfcf1984cb" containerName="cleanup"
Sep 16 21:45:16.597876 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.597694    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c672b65-67d7-4929-b31c-03ce160562df" containerName="cleanup"
Sep 16 21:45:16.600505 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.600484    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:45:16.603468 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.603447    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0de41b\""
Sep 16 21:45:16.603719 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.603701    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0de41b-dockercfg-h6qfr\""
Sep 16 21:45:16.611499 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.611475    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"]
Sep 16 21:45:16.623466 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.623442    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"]
Sep 16 21:45:16.627425 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.627388    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:45:16.636997 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.636943    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"]
Sep 16 21:45:16.666284 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.666250    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deabc723-92ef-472c-a5ae-2a9a3678e012-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf\" (UID: \"deabc723-92ef-472c-a5ae-2a9a3678e012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:45:16.666449 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.666295    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1737eb4-8352-4cad-ac0a-651aecc1ad02-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb\" (UID: \"d1737eb4-8352-4cad-ac0a-651aecc1ad02\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:45:16.666509 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.666489    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xt9r\" (UniqueName: \"kubernetes.io/projected/deabc723-92ef-472c-a5ae-2a9a3678e012-kube-api-access-6xt9r\") pod \"payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf\" (UID: \"deabc723-92ef-472c-a5ae-2a9a3678e012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:45:16.666550 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.666523    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldfq\" (UniqueName: \"kubernetes.io/projected/d1737eb4-8352-4cad-ac0a-651aecc1ad02-kube-api-access-jldfq\") pod \"payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb\" (UID: \"d1737eb4-8352-4cad-ac0a-651aecc1ad02\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:45:16.767451 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.767386    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xt9r\" (UniqueName: \"kubernetes.io/projected/deabc723-92ef-472c-a5ae-2a9a3678e012-kube-api-access-6xt9r\") pod \"payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf\" (UID: \"deabc723-92ef-472c-a5ae-2a9a3678e012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:45:16.767451 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.767453    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldfq\" (UniqueName: \"kubernetes.io/projected/d1737eb4-8352-4cad-ac0a-651aecc1ad02-kube-api-access-jldfq\") pod \"payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb\" (UID: \"d1737eb4-8352-4cad-ac0a-651aecc1ad02\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:45:16.767717 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.767495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deabc723-92ef-472c-a5ae-2a9a3678e012-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf\" (UID: \"deabc723-92ef-472c-a5ae-2a9a3678e012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:45:16.767717 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.767516    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1737eb4-8352-4cad-ac0a-651aecc1ad02-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb\" (UID: \"d1737eb4-8352-4cad-ac0a-651aecc1ad02\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:45:16.768444 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.768419    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1737eb4-8352-4cad-ac0a-651aecc1ad02-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb\" (UID: \"d1737eb4-8352-4cad-ac0a-651aecc1ad02\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:45:16.768502 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.768433    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deabc723-92ef-472c-a5ae-2a9a3678e012-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf\" (UID: \"deabc723-92ef-472c-a5ae-2a9a3678e012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:45:16.775492 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.775465    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xt9r\" (UniqueName: \"kubernetes.io/projected/deabc723-92ef-472c-a5ae-2a9a3678e012-kube-api-access-6xt9r\") pod \"payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf\" (UID: \"deabc723-92ef-472c-a5ae-2a9a3678e012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:45:16.775602 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.775550    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldfq\" (UniqueName: \"kubernetes.io/projected/d1737eb4-8352-4cad-ac0a-651aecc1ad02-kube-api-access-jldfq\") pod \"payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb\" (UID: \"d1737eb4-8352-4cad-ac0a-651aecc1ad02\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:45:16.913446 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.913354    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:45:16.941632 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:16.941600    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:45:17.198880 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:17.198856    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"]
Sep 16 21:45:17.199900 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:45:17.199873    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1737eb4_8352_4cad_ac0a_651aecc1ad02.slice/crio-8be411daa80eaa7809ae24b36a924a815a7112c60b108b07d202308194b31108 WatchSource:0}: Error finding container 8be411daa80eaa7809ae24b36a924a815a7112c60b108b07d202308194b31108: Status 404 returned error can't find the container with id 8be411daa80eaa7809ae24b36a924a815a7112c60b108b07d202308194b31108
Sep 16 21:45:17.213627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:17.213602    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb" event={"ID":"d1737eb4-8352-4cad-ac0a-651aecc1ad02","Type":"ContainerStarted","Data":"8be411daa80eaa7809ae24b36a924a815a7112c60b108b07d202308194b31108"}
Sep 16 21:45:17.230622 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:17.230589    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"]
Sep 16 21:45:17.231272 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:45:17.231197    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeabc723_92ef_472c_a5ae_2a9a3678e012.slice/crio-5cd9f2429741bb6f877210bdee182b19f7665e45edb59a1c6b3ba7026653450d WatchSource:0}: Error finding container 5cd9f2429741bb6f877210bdee182b19f7665e45edb59a1c6b3ba7026653450d: Status 404 returned error can't find the container with id 5cd9f2429741bb6f877210bdee182b19f7665e45edb59a1c6b3ba7026653450d
Sep 16 21:45:18.221453 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:18.221387    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf" event={"ID":"deabc723-92ef-472c-a5ae-2a9a3678e012","Type":"ContainerStarted","Data":"2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51"}
Sep 16 21:45:18.221953 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:18.221476    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf" event={"ID":"deabc723-92ef-472c-a5ae-2a9a3678e012","Type":"ContainerStarted","Data":"5cd9f2429741bb6f877210bdee182b19f7665e45edb59a1c6b3ba7026653450d"}
Sep 16 21:45:18.221953 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:18.221661    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:45:18.223298 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:18.223269    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb" event={"ID":"d1737eb4-8352-4cad-ac0a-651aecc1ad02","Type":"ContainerStarted","Data":"583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e"}
Sep 16 21:45:18.223469 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:18.223319    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:45:18.240339 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:18.240282    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf" podStartSLOduration=2.240262746 podStartE2EDuration="2.240262746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:18.235941616 +0000 UTC m=+1357.621624744" watchObservedRunningTime="2026-09-16 21:45:18.240262746 +0000 UTC m=+1357.625945839"
Sep 16 21:45:18.252592 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:18.252544    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb" podStartSLOduration=2.2525298879999998 podStartE2EDuration="2.252529888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:18.251340606 +0000 UTC m=+1357.637023716" watchObservedRunningTime="2026-09-16 21:45:18.252529888 +0000 UTC m=+1357.638213017"
Sep 16 21:45:29.235074 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:29.235040    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:45:29.235600 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:29.235241    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:45:30.705724 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:30.705686    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:45:47.082347 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.082314    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"]
Sep 16 21:45:47.085725 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.085709    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:45:47.088097 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.088078    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0102fa\""
Sep 16 21:45:47.097978 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.097953    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"]
Sep 16 21:45:47.124662 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.124639    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"]
Sep 16 21:45:47.128719 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.128700    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:45:47.130905 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.130886    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0102fa-dockercfg-zj8kc\""
Sep 16 21:45:47.135129 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.135109    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"]
Sep 16 21:45:47.156731 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.156710    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048635a3-c291-4287-8e3b-0fc3d0539688-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6\" (UID: \"048635a3-c291-4287-8e3b-0fc3d0539688\") " pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:45:47.156821 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.156774    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxcfn\" (UniqueName: \"kubernetes.io/projected/048635a3-c291-4287-8e3b-0fc3d0539688-kube-api-access-mxcfn\") pod \"payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6\" (UID: \"048635a3-c291-4287-8e3b-0fc3d0539688\") " pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:45:47.156821 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.156801    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37fadcd3-04b4-4594-8af3-06abbd57efc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b\" (UID: \"37fadcd3-04b4-4594-8af3-06abbd57efc8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:45:47.156896 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.156863    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlf6m\" (UniqueName: \"kubernetes.io/projected/37fadcd3-04b4-4594-8af3-06abbd57efc8-kube-api-access-dlf6m\") pod \"payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b\" (UID: \"37fadcd3-04b4-4594-8af3-06abbd57efc8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:45:47.257371 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.257328    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37fadcd3-04b4-4594-8af3-06abbd57efc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b\" (UID: \"37fadcd3-04b4-4594-8af3-06abbd57efc8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:45:47.257543 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.257479    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlf6m\" (UniqueName: \"kubernetes.io/projected/37fadcd3-04b4-4594-8af3-06abbd57efc8-kube-api-access-dlf6m\") pod \"payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b\" (UID: \"37fadcd3-04b4-4594-8af3-06abbd57efc8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:45:47.257543 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.257534    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048635a3-c291-4287-8e3b-0fc3d0539688-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6\" (UID: \"048635a3-c291-4287-8e3b-0fc3d0539688\") " pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:45:47.257652 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.257636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxcfn\" (UniqueName: \"kubernetes.io/projected/048635a3-c291-4287-8e3b-0fc3d0539688-kube-api-access-mxcfn\") pod \"payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6\" (UID: \"048635a3-c291-4287-8e3b-0fc3d0539688\") " pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:45:47.258077 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.258052    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37fadcd3-04b4-4594-8af3-06abbd57efc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b\" (UID: \"37fadcd3-04b4-4594-8af3-06abbd57efc8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:45:47.258266 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.258244    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048635a3-c291-4287-8e3b-0fc3d0539688-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6\" (UID: \"048635a3-c291-4287-8e3b-0fc3d0539688\") " pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:45:47.266116 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.266085    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlf6m\" (UniqueName: \"kubernetes.io/projected/37fadcd3-04b4-4594-8af3-06abbd57efc8-kube-api-access-dlf6m\") pod \"payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b\" (UID: \"37fadcd3-04b4-4594-8af3-06abbd57efc8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:45:47.266257 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.266240    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxcfn\" (UniqueName: \"kubernetes.io/projected/048635a3-c291-4287-8e3b-0fc3d0539688-kube-api-access-mxcfn\") pod \"payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6\" (UID: \"048635a3-c291-4287-8e3b-0fc3d0539688\") " pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:45:47.396364 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.396284    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:45:47.440162 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.440128    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:45:47.673731 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.673705    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"]
Sep 16 21:45:47.674921 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:45:47.674889    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fadcd3_04b4_4594_8af3_06abbd57efc8.slice/crio-2eae778a4bd95969b96fd44fdd911a01b1f4be1988a73e09e993b4ed3c448cc7 WatchSource:0}: Error finding container 2eae778a4bd95969b96fd44fdd911a01b1f4be1988a73e09e993b4ed3c448cc7: Status 404 returned error can't find the container with id 2eae778a4bd95969b96fd44fdd911a01b1f4be1988a73e09e993b4ed3c448cc7
Sep 16 21:45:47.726536 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:47.726509    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"]
Sep 16 21:45:47.728623 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:45:47.728598    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048635a3_c291_4287_8e3b_0fc3d0539688.slice/crio-46b9ee0000f33b093c566530231675dbb622ab73c7168dcb5f9381fe91b4f0d2 WatchSource:0}: Error finding container 46b9ee0000f33b093c566530231675dbb622ab73c7168dcb5f9381fe91b4f0d2: Status 404 returned error can't find the container with id 46b9ee0000f33b093c566530231675dbb622ab73c7168dcb5f9381fe91b4f0d2
Sep 16 21:45:48.430636 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:48.430590    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b" event={"ID":"37fadcd3-04b4-4594-8af3-06abbd57efc8","Type":"ContainerStarted","Data":"92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50"}
Sep 16 21:45:48.431097 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:48.430776    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b" event={"ID":"37fadcd3-04b4-4594-8af3-06abbd57efc8","Type":"ContainerStarted","Data":"2eae778a4bd95969b96fd44fdd911a01b1f4be1988a73e09e993b4ed3c448cc7"}
Sep 16 21:45:48.431097 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:48.430820    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:45:48.432453 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:48.432384    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6" event={"ID":"048635a3-c291-4287-8e3b-0fc3d0539688","Type":"ContainerStarted","Data":"1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6"}
Sep 16 21:45:48.432577 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:48.432473    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:45:48.432577 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:48.432490    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6" event={"ID":"048635a3-c291-4287-8e3b-0fc3d0539688","Type":"ContainerStarted","Data":"46b9ee0000f33b093c566530231675dbb622ab73c7168dcb5f9381fe91b4f0d2"}
Sep 16 21:45:48.447779 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:48.447732    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b" podStartSLOduration=1.447718562 podStartE2EDuration="1.447718562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:48.44484436 +0000 UTC m=+1387.830527443" watchObservedRunningTime="2026-09-16 21:45:48.447718562 +0000 UTC m=+1387.833401698"
Sep 16 21:45:48.459472 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:48.459430    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6" podStartSLOduration=1.45939113 podStartE2EDuration="1.45939113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:48.459176093 +0000 UTC m=+1387.844859205" watchObservedRunningTime="2026-09-16 21:45:48.45939113 +0000 UTC m=+1387.845074214"
Sep 16 21:45:58.870167 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:58.870131    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"]
Sep 16 21:45:58.870688 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:58.870350    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw" podUID="725c29ad-36fa-4062-b369-4e2ee78500d7" containerName="payload-processing" containerID="cri-o://fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37" gracePeriod=30
Sep 16 21:45:58.876791 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:58.876761    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"]
Sep 16 21:45:58.877066 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:58.877022    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8" podUID="c78fcf47-01b0-4506-9115-5ec7eefaf5e1" containerName="payload-pre-processing" containerID="cri-o://5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8" gracePeriod=30
Sep 16 21:45:59.178109 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.178080    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:45:59.180149 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.180130    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:45:59.267047 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.267017    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-plugins-config-volume\") pod \"c78fcf47-01b0-4506-9115-5ec7eefaf5e1\" (UID: \"c78fcf47-01b0-4506-9115-5ec7eefaf5e1\") "
Sep 16 21:45:59.267203 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.267063    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwmbc\" (UniqueName: \"kubernetes.io/projected/725c29ad-36fa-4062-b369-4e2ee78500d7-kube-api-access-fwmbc\") pod \"725c29ad-36fa-4062-b369-4e2ee78500d7\" (UID: \"725c29ad-36fa-4062-b369-4e2ee78500d7\") "
Sep 16 21:45:59.267203 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.267146    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725c29ad-36fa-4062-b369-4e2ee78500d7-plugins-config-volume\") pod \"725c29ad-36fa-4062-b369-4e2ee78500d7\" (UID: \"725c29ad-36fa-4062-b369-4e2ee78500d7\") "
Sep 16 21:45:59.267203 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.267176    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttpbm\" (UniqueName: \"kubernetes.io/projected/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-kube-api-access-ttpbm\") pod \"c78fcf47-01b0-4506-9115-5ec7eefaf5e1\" (UID: \"c78fcf47-01b0-4506-9115-5ec7eefaf5e1\") "
Sep 16 21:45:59.267393 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.267366    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c78fcf47-01b0-4506-9115-5ec7eefaf5e1" (UID: "c78fcf47-01b0-4506-9115-5ec7eefaf5e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:45:59.267497 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.267482    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725c29ad-36fa-4062-b369-4e2ee78500d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "725c29ad-36fa-4062-b369-4e2ee78500d7" (UID: "725c29ad-36fa-4062-b369-4e2ee78500d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:45:59.267559 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.267502    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.269253 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.269223    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725c29ad-36fa-4062-b369-4e2ee78500d7-kube-api-access-fwmbc" (OuterVolumeSpecName: "kube-api-access-fwmbc") pod "725c29ad-36fa-4062-b369-4e2ee78500d7" (UID: "725c29ad-36fa-4062-b369-4e2ee78500d7"). InnerVolumeSpecName "kube-api-access-fwmbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:45:59.269372 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.269285    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-kube-api-access-ttpbm" (OuterVolumeSpecName: "kube-api-access-ttpbm") pod "c78fcf47-01b0-4506-9115-5ec7eefaf5e1" (UID: "c78fcf47-01b0-4506-9115-5ec7eefaf5e1"). InnerVolumeSpecName "kube-api-access-ttpbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:45:59.368155 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.368118    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725c29ad-36fa-4062-b369-4e2ee78500d7-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.368155 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.368151    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttpbm\" (UniqueName: \"kubernetes.io/projected/c78fcf47-01b0-4506-9115-5ec7eefaf5e1-kube-api-access-ttpbm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.368155 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.368161    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwmbc\" (UniqueName: \"kubernetes.io/projected/725c29ad-36fa-4062-b369-4e2ee78500d7-kube-api-access-fwmbc\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.440924 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.440848    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:45:59.441050 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.440934    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:45:59.509974 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.509937    2567 generic.go:356] "Generic (PLEG): container finished" podID="725c29ad-36fa-4062-b369-4e2ee78500d7" containerID="fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37" exitCode=0
Sep 16 21:45:59.509974 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.509979    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"
Sep 16 21:45:59.510217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.510016    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw" event={"ID":"725c29ad-36fa-4062-b369-4e2ee78500d7","Type":"ContainerDied","Data":"fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37"}
Sep 16 21:45:59.510217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.510067    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw" event={"ID":"725c29ad-36fa-4062-b369-4e2ee78500d7","Type":"ContainerDied","Data":"5518ad395f1db5261305aa99961ee96bc6d92b52b142921513529b80e6a6c37d"}
Sep 16 21:45:59.510217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.510098    2567 scope.go:117] "RemoveContainer" containerID="fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37"
Sep 16 21:45:59.511588 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.511557    2567 generic.go:356] "Generic (PLEG): container finished" podID="c78fcf47-01b0-4506-9115-5ec7eefaf5e1" containerID="5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8" exitCode=0
Sep 16 21:45:59.511712 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.511611    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"
Sep 16 21:45:59.511712 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.511648    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8" event={"ID":"c78fcf47-01b0-4506-9115-5ec7eefaf5e1","Type":"ContainerDied","Data":"5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8"}
Sep 16 21:45:59.511837 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.511710    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8" event={"ID":"c78fcf47-01b0-4506-9115-5ec7eefaf5e1","Type":"ContainerDied","Data":"c789dd94cf134cc490e402eaa60ed7f43c27c37c50402e2abf6e80ac80922884"}
Sep 16 21:45:59.521448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.521429    2567 scope.go:117] "RemoveContainer" containerID="fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37"
Sep 16 21:45:59.521927 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:45:59.521903    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37\": container with ID starting with fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37 not found: ID does not exist" containerID="fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37"
Sep 16 21:45:59.522047 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.521936    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37"} err="failed to get container status \"fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37\": rpc error: code = NotFound desc = could not find container \"fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37\": container with ID starting with fa8e20b37f415e94d81a70c324af94b626d12c843a73823fe3556129e885ba37 not found: ID does not exist"
Sep 16 21:45:59.522047 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.521958    2567 scope.go:117] "RemoveContainer" containerID="5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8"
Sep 16 21:45:59.534550 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.534531    2567 scope.go:117] "RemoveContainer" containerID="5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8"
Sep 16 21:45:59.534854 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:45:59.534831    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8\": container with ID starting with 5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8 not found: ID does not exist" containerID="5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8"
Sep 16 21:45:59.534934 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.534865    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8"} err="failed to get container status \"5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8\": rpc error: code = NotFound desc = could not find container \"5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8\": container with ID starting with 5b04ff25eba0f734c31ffd9c1fdd3d0e24e2c66590ea6975f46d77f570e23aa8 not found: ID does not exist"
Sep 16 21:45:59.535090 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.535076    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"]
Sep 16 21:45:59.539205 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.539185    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-35f5fd-d979cbb8f-7lkkw"]
Sep 16 21:45:59.543549 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.543492    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"]
Sep 16 21:45:59.549142 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:45:59.549121    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-35f5fd-757d78b8c8-hpqv8"]
Sep 16 21:46:01.109713 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:01.109682    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="725c29ad-36fa-4062-b369-4e2ee78500d7" path="/var/lib/kubelet/pods/725c29ad-36fa-4062-b369-4e2ee78500d7/volumes"
Sep 16 21:46:01.110075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:01.110059    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c78fcf47-01b0-4506-9115-5ec7eefaf5e1" path="/var/lib/kubelet/pods/c78fcf47-01b0-4506-9115-5ec7eefaf5e1/volumes"
Sep 16 21:46:11.889465 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:11.889428    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:46:23.348038 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.347983    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"]
Sep 16 21:46:23.348533 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.348266    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb" podUID="d1737eb4-8352-4cad-ac0a-651aecc1ad02" containerName="payload-processing" containerID="cri-o://583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e" gracePeriod=30
Sep 16 21:46:23.354368 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.354338    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"]
Sep 16 21:46:23.354816 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.354762    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf" podUID="deabc723-92ef-472c-a5ae-2a9a3678e012" containerName="payload-pre-processing" containerID="cri-o://2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51" gracePeriod=30
Sep 16 21:46:23.640665 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.640628    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:46:23.643862 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.643840    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:46:23.681179 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.680754    2567 generic.go:356] "Generic (PLEG): container finished" podID="deabc723-92ef-472c-a5ae-2a9a3678e012" containerID="2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51" exitCode=0
Sep 16 21:46:23.681179 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.680885    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf" event={"ID":"deabc723-92ef-472c-a5ae-2a9a3678e012","Type":"ContainerDied","Data":"2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51"}
Sep 16 21:46:23.681179 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.680947    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf" event={"ID":"deabc723-92ef-472c-a5ae-2a9a3678e012","Type":"ContainerDied","Data":"5cd9f2429741bb6f877210bdee182b19f7665e45edb59a1c6b3ba7026653450d"}
Sep 16 21:46:23.681179 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.680978    2567 scope.go:117] "RemoveContainer" containerID="2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51"
Sep 16 21:46:23.681528 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.681213    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"
Sep 16 21:46:23.682843 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.682816    2567 generic.go:356] "Generic (PLEG): container finished" podID="d1737eb4-8352-4cad-ac0a-651aecc1ad02" containerID="583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e" exitCode=0
Sep 16 21:46:23.682961 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.682928    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"
Sep 16 21:46:23.683036 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.682957    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb" event={"ID":"d1737eb4-8352-4cad-ac0a-651aecc1ad02","Type":"ContainerDied","Data":"583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e"}
Sep 16 21:46:23.683036 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.683011    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb" event={"ID":"d1737eb4-8352-4cad-ac0a-651aecc1ad02","Type":"ContainerDied","Data":"8be411daa80eaa7809ae24b36a924a815a7112c60b108b07d202308194b31108"}
Sep 16 21:46:23.691165 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.691141    2567 scope.go:117] "RemoveContainer" containerID="2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51"
Sep 16 21:46:23.691465 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:46:23.691445    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51\": container with ID starting with 2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51 not found: ID does not exist" containerID="2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51"
Sep 16 21:46:23.691525 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.691475    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51"} err="failed to get container status \"2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51\": rpc error: code = NotFound desc = could not find container \"2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51\": container with ID starting with 2a90310d2c7f1618daf11a97ab6daff197ff6057ec210264c65cc40ca7d19d51 not found: ID does not exist"
Sep 16 21:46:23.691525 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.691492    2567 scope.go:117] "RemoveContainer" containerID="583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e"
Sep 16 21:46:23.700875 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.700854    2567 scope.go:117] "RemoveContainer" containerID="583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e"
Sep 16 21:46:23.701176 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:46:23.701156    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e\": container with ID starting with 583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e not found: ID does not exist" containerID="583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e"
Sep 16 21:46:23.701241 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.701183    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e"} err="failed to get container status \"583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e\": rpc error: code = NotFound desc = could not find container \"583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e\": container with ID starting with 583555f4b89ed5b8e9e4fac010b2c6e70a130de6505d658b9b067f6981bda56e not found: ID does not exist"
Sep 16 21:46:23.796471 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.796422    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xt9r\" (UniqueName: \"kubernetes.io/projected/deabc723-92ef-472c-a5ae-2a9a3678e012-kube-api-access-6xt9r\") pod \"deabc723-92ef-472c-a5ae-2a9a3678e012\" (UID: \"deabc723-92ef-472c-a5ae-2a9a3678e012\") "
Sep 16 21:46:23.796658 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.796521    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deabc723-92ef-472c-a5ae-2a9a3678e012-plugins-config-volume\") pod \"deabc723-92ef-472c-a5ae-2a9a3678e012\" (UID: \"deabc723-92ef-472c-a5ae-2a9a3678e012\") "
Sep 16 21:46:23.796658 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.796552    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1737eb4-8352-4cad-ac0a-651aecc1ad02-plugins-config-volume\") pod \"d1737eb4-8352-4cad-ac0a-651aecc1ad02\" (UID: \"d1737eb4-8352-4cad-ac0a-651aecc1ad02\") "
Sep 16 21:46:23.796658 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.796586    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jldfq\" (UniqueName: \"kubernetes.io/projected/d1737eb4-8352-4cad-ac0a-651aecc1ad02-kube-api-access-jldfq\") pod \"d1737eb4-8352-4cad-ac0a-651aecc1ad02\" (UID: \"d1737eb4-8352-4cad-ac0a-651aecc1ad02\") "
Sep 16 21:46:23.796916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.796882    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1737eb4-8352-4cad-ac0a-651aecc1ad02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1737eb4-8352-4cad-ac0a-651aecc1ad02" (UID: "d1737eb4-8352-4cad-ac0a-651aecc1ad02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:23.796916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.796891    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deabc723-92ef-472c-a5ae-2a9a3678e012-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "deabc723-92ef-472c-a5ae-2a9a3678e012" (UID: "deabc723-92ef-472c-a5ae-2a9a3678e012"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:23.798659 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.798638    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deabc723-92ef-472c-a5ae-2a9a3678e012-kube-api-access-6xt9r" (OuterVolumeSpecName: "kube-api-access-6xt9r") pod "deabc723-92ef-472c-a5ae-2a9a3678e012" (UID: "deabc723-92ef-472c-a5ae-2a9a3678e012"). InnerVolumeSpecName "kube-api-access-6xt9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:23.798716 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.798668    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1737eb4-8352-4cad-ac0a-651aecc1ad02-kube-api-access-jldfq" (OuterVolumeSpecName: "kube-api-access-jldfq") pod "d1737eb4-8352-4cad-ac0a-651aecc1ad02" (UID: "d1737eb4-8352-4cad-ac0a-651aecc1ad02"). InnerVolumeSpecName "kube-api-access-jldfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:23.897937 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.897844    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deabc723-92ef-472c-a5ae-2a9a3678e012-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:46:23.897937 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.897884    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1737eb4-8352-4cad-ac0a-651aecc1ad02-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:46:23.897937 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.897898    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jldfq\" (UniqueName: \"kubernetes.io/projected/d1737eb4-8352-4cad-ac0a-651aecc1ad02-kube-api-access-jldfq\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:46:23.897937 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:23.897909    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xt9r\" (UniqueName: \"kubernetes.io/projected/deabc723-92ef-472c-a5ae-2a9a3678e012-kube-api-access-6xt9r\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:46:24.002641 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:24.002600    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"]
Sep 16 21:46:24.008235 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:24.008214    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de41b-b766857bb-p5fdf"]
Sep 16 21:46:24.018226 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:24.018202    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"]
Sep 16 21:46:24.023593 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:24.023570    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de41b-5c964d5cd7-r77bb"]
Sep 16 21:46:25.108944 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:25.108908    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1737eb4-8352-4cad-ac0a-651aecc1ad02" path="/var/lib/kubelet/pods/d1737eb4-8352-4cad-ac0a-651aecc1ad02/volumes"
Sep 16 21:46:25.109461 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:25.109434    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deabc723-92ef-472c-a5ae-2a9a3678e012" path="/var/lib/kubelet/pods/deabc723-92ef-472c-a5ae-2a9a3678e012/volumes"
Sep 16 21:46:37.787307 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:37.787231    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:46:41.350087 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:41.350052    2567 scope.go:117] "RemoveContainer" containerID="109b6abaabbb4c5b407d9b9cf2b842c8ebf535666827ac6e867e3d78a5c887fa"
Sep 16 21:46:45.942043 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:45.942009    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"]
Sep 16 21:46:45.942526 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:45.942238    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6" podUID="048635a3-c291-4287-8e3b-0fc3d0539688" containerName="payload-processing" containerID="cri-o://1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6" gracePeriod=30
Sep 16 21:46:45.959228 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:45.958715    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"]
Sep 16 21:46:45.959575 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:45.959544    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b" podUID="37fadcd3-04b4-4594-8af3-06abbd57efc8" containerName="payload-pre-processing" containerID="cri-o://92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50" gracePeriod=30
Sep 16 21:46:46.251046 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.251019    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:46:46.253723 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.253699    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:46:46.309184 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.309153    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048635a3-c291-4287-8e3b-0fc3d0539688-plugins-config-volume\") pod \"048635a3-c291-4287-8e3b-0fc3d0539688\" (UID: \"048635a3-c291-4287-8e3b-0fc3d0539688\") "
Sep 16 21:46:46.309379 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.309204    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxcfn\" (UniqueName: \"kubernetes.io/projected/048635a3-c291-4287-8e3b-0fc3d0539688-kube-api-access-mxcfn\") pod \"048635a3-c291-4287-8e3b-0fc3d0539688\" (UID: \"048635a3-c291-4287-8e3b-0fc3d0539688\") "
Sep 16 21:46:46.309379 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.309259    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlf6m\" (UniqueName: \"kubernetes.io/projected/37fadcd3-04b4-4594-8af3-06abbd57efc8-kube-api-access-dlf6m\") pod \"37fadcd3-04b4-4594-8af3-06abbd57efc8\" (UID: \"37fadcd3-04b4-4594-8af3-06abbd57efc8\") "
Sep 16 21:46:46.309379 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.309293    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37fadcd3-04b4-4594-8af3-06abbd57efc8-plugins-config-volume\") pod \"37fadcd3-04b4-4594-8af3-06abbd57efc8\" (UID: \"37fadcd3-04b4-4594-8af3-06abbd57efc8\") "
Sep 16 21:46:46.309678 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.309652    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048635a3-c291-4287-8e3b-0fc3d0539688-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "048635a3-c291-4287-8e3b-0fc3d0539688" (UID: "048635a3-c291-4287-8e3b-0fc3d0539688"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:46.309757 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.309731    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fadcd3-04b4-4594-8af3-06abbd57efc8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37fadcd3-04b4-4594-8af3-06abbd57efc8" (UID: "37fadcd3-04b4-4594-8af3-06abbd57efc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:46.311430 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.311383    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048635a3-c291-4287-8e3b-0fc3d0539688-kube-api-access-mxcfn" (OuterVolumeSpecName: "kube-api-access-mxcfn") pod "048635a3-c291-4287-8e3b-0fc3d0539688" (UID: "048635a3-c291-4287-8e3b-0fc3d0539688"). InnerVolumeSpecName "kube-api-access-mxcfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:46.311766 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.311742    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fadcd3-04b4-4594-8af3-06abbd57efc8-kube-api-access-dlf6m" (OuterVolumeSpecName: "kube-api-access-dlf6m") pod "37fadcd3-04b4-4594-8af3-06abbd57efc8" (UID: "37fadcd3-04b4-4594-8af3-06abbd57efc8"). InnerVolumeSpecName "kube-api-access-dlf6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:46.410329 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.410297    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlf6m\" (UniqueName: \"kubernetes.io/projected/37fadcd3-04b4-4594-8af3-06abbd57efc8-kube-api-access-dlf6m\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:46:46.410329 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.410324    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37fadcd3-04b4-4594-8af3-06abbd57efc8-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:46:46.410329 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.410333    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048635a3-c291-4287-8e3b-0fc3d0539688-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:46:46.410572 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.410342    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxcfn\" (UniqueName: \"kubernetes.io/projected/048635a3-c291-4287-8e3b-0fc3d0539688-kube-api-access-mxcfn\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:46:46.846405 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.846356    2567 generic.go:356] "Generic (PLEG): container finished" podID="048635a3-c291-4287-8e3b-0fc3d0539688" containerID="1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6" exitCode=0
Sep 16 21:46:46.846585 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.846411    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"
Sep 16 21:46:46.846585 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.846427    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6" event={"ID":"048635a3-c291-4287-8e3b-0fc3d0539688","Type":"ContainerDied","Data":"1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6"}
Sep 16 21:46:46.846585 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.846482    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6" event={"ID":"048635a3-c291-4287-8e3b-0fc3d0539688","Type":"ContainerDied","Data":"46b9ee0000f33b093c566530231675dbb622ab73c7168dcb5f9381fe91b4f0d2"}
Sep 16 21:46:46.846585 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.846505    2567 scope.go:117] "RemoveContainer" containerID="1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6"
Sep 16 21:46:46.848079 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.848052    2567 generic.go:356] "Generic (PLEG): container finished" podID="37fadcd3-04b4-4594-8af3-06abbd57efc8" containerID="92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50" exitCode=0
Sep 16 21:46:46.848218 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.848121    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"
Sep 16 21:46:46.848218 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.848141    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b" event={"ID":"37fadcd3-04b4-4594-8af3-06abbd57efc8","Type":"ContainerDied","Data":"92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50"}
Sep 16 21:46:46.848218 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.848181    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b" event={"ID":"37fadcd3-04b4-4594-8af3-06abbd57efc8","Type":"ContainerDied","Data":"2eae778a4bd95969b96fd44fdd911a01b1f4be1988a73e09e993b4ed3c448cc7"}
Sep 16 21:46:46.856786 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.856768    2567 scope.go:117] "RemoveContainer" containerID="1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6"
Sep 16 21:46:46.857096 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:46:46.857077    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6\": container with ID starting with 1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6 not found: ID does not exist" containerID="1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6"
Sep 16 21:46:46.857188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.857104    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6"} err="failed to get container status \"1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6\": rpc error: code = NotFound desc = could not find container \"1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6\": container with ID starting with 1b9fd10ffab9862d6e5965878fd3d0c551797fa48b4f639a66f558a0725345d6 not found: ID does not exist"
Sep 16 21:46:46.857188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.857120    2567 scope.go:117] "RemoveContainer" containerID="92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50"
Sep 16 21:46:46.865582 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.865565    2567 scope.go:117] "RemoveContainer" containerID="92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50"
Sep 16 21:46:46.865816 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:46:46.865801    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50\": container with ID starting with 92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50 not found: ID does not exist" containerID="92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50"
Sep 16 21:46:46.865865 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.865821    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50"} err="failed to get container status \"92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50\": rpc error: code = NotFound desc = could not find container \"92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50\": container with ID starting with 92e18abca2924f9809d11eab1d6cf7763fc520aecfb0c0624ab6369b9172be50 not found: ID does not exist"
Sep 16 21:46:46.870193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.870172    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"]
Sep 16 21:46:46.877741 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.877721    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0102fa-6f8c658957-4fgf6"]
Sep 16 21:46:46.890210 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.890184    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"]
Sep 16 21:46:46.892799 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:46.892782    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0102fa-77945f78f4-x7q5b"]
Sep 16 21:46:47.109720 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:47.109650    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="048635a3-c291-4287-8e3b-0fc3d0539688" path="/var/lib/kubelet/pods/048635a3-c291-4287-8e3b-0fc3d0539688/volumes"
Sep 16 21:46:47.110085 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:46:47.109996    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37fadcd3-04b4-4594-8af3-06abbd57efc8" path="/var/lib/kubelet/pods/37fadcd3-04b4-4594-8af3-06abbd57efc8/volumes"
Sep 16 21:47:04.135659 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.135623    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc"]
Sep 16 21:47:04.141981 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.141943    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-296002-29826585-8k2hc"]
Sep 16 21:47:04.170407 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.170360    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"]
Sep 16 21:47:04.170679 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.170631    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg" podUID="de1b7dcd-d05f-4449-9618-1bd9f23a6e0a" containerName="payload-processing" containerID="cri-o://40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e" gracePeriod=30
Sep 16 21:47:04.179188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.178570    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"]
Sep 16 21:47:04.179188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.178779    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx" podUID="d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa" containerName="payload-pre-processing" containerID="cri-o://505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b" gracePeriod=30
Sep 16 21:47:04.445859 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.445823    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:47:04.461190 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.461166    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:47:04.577808 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.577774    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p742b\" (UniqueName: \"kubernetes.io/projected/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-kube-api-access-p742b\") pod \"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a\" (UID: \"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a\") "
Sep 16 21:47:04.577977 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.577817    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-plugins-config-volume\") pod \"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a\" (UID: \"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a\") "
Sep 16 21:47:04.577977 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.577876    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-plugins-config-volume\") pod \"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa\" (UID: \"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa\") "
Sep 16 21:47:04.577977 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.577948    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm8x9\" (UniqueName: \"kubernetes.io/projected/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-kube-api-access-tm8x9\") pod \"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa\" (UID: \"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa\") "
Sep 16 21:47:04.578205 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.578179    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de1b7dcd-d05f-4449-9618-1bd9f23a6e0a" (UID: "de1b7dcd-d05f-4449-9618-1bd9f23a6e0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:04.578313 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.578294    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa" (UID: "d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:04.579862 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.579839    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-kube-api-access-p742b" (OuterVolumeSpecName: "kube-api-access-p742b") pod "de1b7dcd-d05f-4449-9618-1bd9f23a6e0a" (UID: "de1b7dcd-d05f-4449-9618-1bd9f23a6e0a"). InnerVolumeSpecName "kube-api-access-p742b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:04.580039 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.580021    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-kube-api-access-tm8x9" (OuterVolumeSpecName: "kube-api-access-tm8x9") pod "d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa" (UID: "d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa"). InnerVolumeSpecName "kube-api-access-tm8x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:04.679335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.679256    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm8x9\" (UniqueName: \"kubernetes.io/projected/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-kube-api-access-tm8x9\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:04.679335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.679286    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p742b\" (UniqueName: \"kubernetes.io/projected/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-kube-api-access-p742b\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:04.679335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.679296    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:04.679335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.679304    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:04.974343 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.974306    2567 generic.go:356] "Generic (PLEG): container finished" podID="de1b7dcd-d05f-4449-9618-1bd9f23a6e0a" containerID="40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e" exitCode=0
Sep 16 21:47:04.974546 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.974350    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"
Sep 16 21:47:04.974546 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.974389    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg" event={"ID":"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a","Type":"ContainerDied","Data":"40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e"}
Sep 16 21:47:04.974546 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.974462    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg" event={"ID":"de1b7dcd-d05f-4449-9618-1bd9f23a6e0a","Type":"ContainerDied","Data":"a68082b8bdc50f46cbb69c16380e2703db1f9e264bbc89d2458b85c67bf1c6a1"}
Sep 16 21:47:04.974546 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.974490    2567 scope.go:117] "RemoveContainer" containerID="40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e"
Sep 16 21:47:04.975891 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.975865    2567 generic.go:356] "Generic (PLEG): container finished" podID="d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa" containerID="505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b" exitCode=0
Sep 16 21:47:04.975997 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.975912    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"
Sep 16 21:47:04.976041 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.975914    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx" event={"ID":"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa","Type":"ContainerDied","Data":"505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b"}
Sep 16 21:47:04.976093 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.976038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx" event={"ID":"d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa","Type":"ContainerDied","Data":"752b32739cc348e14f008c6d52d51e6f5d63d799e97e9293dcfac96018b716dd"}
Sep 16 21:47:04.986120 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.986105    2567 scope.go:117] "RemoveContainer" containerID="40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e"
Sep 16 21:47:04.986551 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:47:04.986530    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e\": container with ID starting with 40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e not found: ID does not exist" containerID="40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e"
Sep 16 21:47:04.986623 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.986559    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e"} err="failed to get container status \"40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e\": rpc error: code = NotFound desc = could not find container \"40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e\": container with ID starting with 40ec0fc8a08450fce322c44ebd4c743b3b360b4f4934bf7d5f3c9d7051e16e3e not found: ID does not exist"
Sep 16 21:47:04.986623 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.986575    2567 scope.go:117] "RemoveContainer" containerID="505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b"
Sep 16 21:47:04.996036 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.996017    2567 scope.go:117] "RemoveContainer" containerID="505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b"
Sep 16 21:47:04.996298 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:47:04.996279    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b\": container with ID starting with 505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b not found: ID does not exist" containerID="505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b"
Sep 16 21:47:04.996368 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.996303    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b"} err="failed to get container status \"505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b\": rpc error: code = NotFound desc = could not find container \"505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b\": container with ID starting with 505e81981f786d3d72f94c3d1b7a466201fc92b9da4b794d083746747bb19a5b not found: ID does not exist"
Sep 16 21:47:04.997728 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:04.997709    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"]
Sep 16 21:47:05.002786 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:05.002765    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-296002-6555c6d555-nk4zg"]
Sep 16 21:47:05.014733 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:05.014709    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"]
Sep 16 21:47:05.019272 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:05.019254    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-296002-55b86c647d-gclqx"]
Sep 16 21:47:05.115243 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:05.115210    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c672b65-67d7-4929-b31c-03ce160562df" path="/var/lib/kubelet/pods/6c672b65-67d7-4929-b31c-03ce160562df/volumes"
Sep 16 21:47:05.115558 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:05.115544    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa" path="/var/lib/kubelet/pods/d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa/volumes"
Sep 16 21:47:05.115867 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:05.115855    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de1b7dcd-d05f-4449-9618-1bd9f23a6e0a" path="/var/lib/kubelet/pods/de1b7dcd-d05f-4449-9618-1bd9f23a6e0a/volumes"
Sep 16 21:47:07.777328 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:07.777282    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:47:20.543692 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:20.543655    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"]
Sep 16 21:47:20.548829 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:20.548801    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"]
Sep 16 21:47:20.557730 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:20.557702    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"]
Sep 16 21:47:21.842065 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:21.842032    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn"]
Sep 16 21:47:21.845607 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:21.844690    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-365ea1-29826585-gv7vn"]
Sep 16 21:47:21.867188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:21.867156    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"]
Sep 16 21:47:21.867372 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:21.867349    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt" podUID="466e6ec5-8ed5-47d4-a566-a009a0fadc4c" containerName="payload-processing" containerID="cri-o://ad5668f20409adc752484e13031dedf5996ce8ca66bfdd9786112082ddef404d" gracePeriod=30
Sep 16 21:47:21.879428 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:21.877681    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"]
Sep 16 21:47:21.883370 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:21.880255    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8" podUID="d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c" containerName="payload-pre-processing" containerID="cri-o://1a5494ceff31c1a54dabe64ec1d575ed056e13e2516f8403266922b4f83d9d82" gracePeriod=30
Sep 16 21:47:22.089541 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.089509    2567 generic.go:356] "Generic (PLEG): container finished" podID="466e6ec5-8ed5-47d4-a566-a009a0fadc4c" containerID="ad5668f20409adc752484e13031dedf5996ce8ca66bfdd9786112082ddef404d" exitCode=0
Sep 16 21:47:22.089720 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.089587    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt" event={"ID":"466e6ec5-8ed5-47d4-a566-a009a0fadc4c","Type":"ContainerDied","Data":"ad5668f20409adc752484e13031dedf5996ce8ca66bfdd9786112082ddef404d"}
Sep 16 21:47:22.091278 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.091252    2567 generic.go:356] "Generic (PLEG): container finished" podID="d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c" containerID="1a5494ceff31c1a54dabe64ec1d575ed056e13e2516f8403266922b4f83d9d82" exitCode=0
Sep 16 21:47:22.091389 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.091322    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8" event={"ID":"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c","Type":"ContainerDied","Data":"1a5494ceff31c1a54dabe64ec1d575ed056e13e2516f8403266922b4f83d9d82"}
Sep 16 21:47:22.151663 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.151638    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:47:22.184912 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.184497    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:47:22.338621 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.338587    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-plugins-config-volume\") pod \"466e6ec5-8ed5-47d4-a566-a009a0fadc4c\" (UID: \"466e6ec5-8ed5-47d4-a566-a009a0fadc4c\") "
Sep 16 21:47:22.338805 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.338663    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx68t\" (UniqueName: \"kubernetes.io/projected/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-kube-api-access-nx68t\") pod \"466e6ec5-8ed5-47d4-a566-a009a0fadc4c\" (UID: \"466e6ec5-8ed5-47d4-a566-a009a0fadc4c\") "
Sep 16 21:47:22.338805 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.338725    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfjvh\" (UniqueName: \"kubernetes.io/projected/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-kube-api-access-vfjvh\") pod \"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c\" (UID: \"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c\") "
Sep 16 21:47:22.338805 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.338786    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-plugins-config-volume\") pod \"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c\" (UID: \"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c\") "
Sep 16 21:47:22.338997 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.338969    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "466e6ec5-8ed5-47d4-a566-a009a0fadc4c" (UID: "466e6ec5-8ed5-47d4-a566-a009a0fadc4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:22.339154 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.339135    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c" (UID: "d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:22.340726 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.340700    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-kube-api-access-nx68t" (OuterVolumeSpecName: "kube-api-access-nx68t") pod "466e6ec5-8ed5-47d4-a566-a009a0fadc4c" (UID: "466e6ec5-8ed5-47d4-a566-a009a0fadc4c"). InnerVolumeSpecName "kube-api-access-nx68t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:22.340832 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.340805    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-kube-api-access-vfjvh" (OuterVolumeSpecName: "kube-api-access-vfjvh") pod "d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c" (UID: "d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c"). InnerVolumeSpecName "kube-api-access-vfjvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:22.440232 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.440156    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:22.440232 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.440184    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:22.440232 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.440193    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx68t\" (UniqueName: \"kubernetes.io/projected/466e6ec5-8ed5-47d4-a566-a009a0fadc4c-kube-api-access-nx68t\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:22.440232 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:22.440208    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfjvh\" (UniqueName: \"kubernetes.io/projected/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c-kube-api-access-vfjvh\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:23.098957 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:23.098931    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"
Sep 16 21:47:23.099372 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:23.098930    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt" event={"ID":"466e6ec5-8ed5-47d4-a566-a009a0fadc4c","Type":"ContainerDied","Data":"132d600a4403fc52bafec5acc69cb8ffac2e302c1f5747ed411462d9206be349"}
Sep 16 21:47:23.099372 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:23.099054    2567 scope.go:117] "RemoveContainer" containerID="ad5668f20409adc752484e13031dedf5996ce8ca66bfdd9786112082ddef404d"
Sep 16 21:47:23.100510 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:23.100492    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8" event={"ID":"d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c","Type":"ContainerDied","Data":"4fa76b00b23262b320a07c618af40efb5a48c9756a79bd9928ee27609a6b0847"}
Sep 16 21:47:23.100510 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:23.100502    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"
Sep 16 21:47:23.111436 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:23.109084    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50fd6533-27b6-4f40-b4c4-ebcfcf1984cb" path="/var/lib/kubelet/pods/50fd6533-27b6-4f40-b4c4-ebcfcf1984cb/volumes"
Sep 16 21:47:23.113989 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:23.113958    2567 scope.go:117] "RemoveContainer" containerID="1a5494ceff31c1a54dabe64ec1d575ed056e13e2516f8403266922b4f83d9d82"
Sep 16 21:47:23.131465 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:23.131444    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"]
Sep 16 21:47:23.135275 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:23.135253    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-365ea1-58bfc7f998-b59h8"]
Sep 16 21:47:23.144116 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:23.144096    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"]
Sep 16 21:47:23.147484 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:23.147464    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-365ea1-574ffb5d85-4xrpt"]
Sep 16 21:47:25.108935 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:25.108903    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="466e6ec5-8ed5-47d4-a566-a009a0fadc4c" path="/var/lib/kubelet/pods/466e6ec5-8ed5-47d4-a566-a009a0fadc4c/volumes"
Sep 16 21:47:25.109361 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:25.109270    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c" path="/var/lib/kubelet/pods/d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c/volumes"
Sep 16 21:47:31.246262 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:31.246224    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"]
Sep 16 21:47:31.246756 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:31.246510    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v" podUID="70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f" containerName="payload-pre-processing" containerID="cri-o://e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46" gracePeriod=30
Sep 16 21:47:31.494971 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:31.494945    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:47:31.516537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:31.516472    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mg4f\" (UniqueName: \"kubernetes.io/projected/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-kube-api-access-5mg4f\") pod \"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f\" (UID: \"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f\") "
Sep 16 21:47:31.516682 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:31.516657    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-plugins-config-volume\") pod \"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f\" (UID: \"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f\") "
Sep 16 21:47:31.517062 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:31.517033    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f" (UID: "70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:31.518565 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:31.518537    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-kube-api-access-5mg4f" (OuterVolumeSpecName: "kube-api-access-5mg4f") pod "70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f" (UID: "70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f"). InnerVolumeSpecName "kube-api-access-5mg4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:31.618500 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:31.618472    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:31.618500 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:31.618497    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mg4f\" (UniqueName: \"kubernetes.io/projected/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f-kube-api-access-5mg4f\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:32.163100 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:32.163065    2567 generic.go:356] "Generic (PLEG): container finished" podID="70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f" containerID="e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46" exitCode=0
Sep 16 21:47:32.163293 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:32.163107    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"
Sep 16 21:47:32.163293 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:32.163150    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v" event={"ID":"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f","Type":"ContainerDied","Data":"e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46"}
Sep 16 21:47:32.163293 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:32.163209    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v" event={"ID":"70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f","Type":"ContainerDied","Data":"6f2101b647b09347ba612317355b6f66e4ce8cfae825112682561b53a5fba7ab"}
Sep 16 21:47:32.163293 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:32.163239    2567 scope.go:117] "RemoveContainer" containerID="e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46"
Sep 16 21:47:32.173453 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:32.173437    2567 scope.go:117] "RemoveContainer" containerID="e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46"
Sep 16 21:47:32.173723 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:47:32.173704    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46\": container with ID starting with e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46 not found: ID does not exist" containerID="e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46"
Sep 16 21:47:32.173784 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:32.173731    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46"} err="failed to get container status \"e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46\": rpc error: code = NotFound desc = could not find container \"e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46\": container with ID starting with e5bb4308a3536eb5c63869e168f43f669ec903f81f4bda4868670f074dc55e46 not found: ID does not exist"
Sep 16 21:47:32.186029 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:32.186005    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"]
Sep 16 21:47:32.190831 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:32.190807    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e047d8-59f5b5fbcd-72h6v"]
Sep 16 21:47:33.108791 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:33.108757    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f" path="/var/lib/kubelet/pods/70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f/volumes"
Sep 16 21:47:33.983289 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:33.983254    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:47:35.564545 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.564507    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd" podUID="d150203a-6382-4b30-94e1-16131d9f64d5" containerName="main" containerID="cri-o://d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882" gracePeriod=45
Sep 16 21:47:35.569448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.569419    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h" podUID="01414087-99a7-46bd-ae45-67ecb110c13c" containerName="main" containerID="cri-o://4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393" gracePeriod=45
Sep 16 21:47:35.577218 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.577187    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz" podUID="8b303241-58d2-4262-be41-db53acad4694" containerName="main" containerID="cri-o://dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb" gracePeriod=45
Sep 16 21:47:35.874539 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.874475    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:47:35.878030 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.878010    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:47:35.882592 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.882572    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:47:35.959448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.959416    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-home\") pod \"01414087-99a7-46bd-ae45-67ecb110c13c\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") "
Sep 16 21:47:35.959627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.959461    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-tmp-dir\") pod \"8b303241-58d2-4262-be41-db53acad4694\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") "
Sep 16 21:47:35.959627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.959503    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh2vr\" (UniqueName: \"kubernetes.io/projected/8b303241-58d2-4262-be41-db53acad4694-kube-api-access-vh2vr\") pod \"8b303241-58d2-4262-be41-db53acad4694\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") "
Sep 16 21:47:35.959627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.959523    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-dshm\") pod \"d150203a-6382-4b30-94e1-16131d9f64d5\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") "
Sep 16 21:47:35.959627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.959541    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-home\") pod \"d150203a-6382-4b30-94e1-16131d9f64d5\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") "
Sep 16 21:47:35.959627 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.959568    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22mqj\" (UniqueName: \"kubernetes.io/projected/01414087-99a7-46bd-ae45-67ecb110c13c-kube-api-access-22mqj\") pod \"01414087-99a7-46bd-ae45-67ecb110c13c\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") "
Sep 16 21:47:35.959896 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.959765    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-home" (OuterVolumeSpecName: "home") pod "01414087-99a7-46bd-ae45-67ecb110c13c" (UID: "01414087-99a7-46bd-ae45-67ecb110c13c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.959896 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.959808    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b303241-58d2-4262-be41-db53acad4694" (UID: "8b303241-58d2-4262-be41-db53acad4694"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.960005 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.959953    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-model-cache\") pod \"8b303241-58d2-4262-be41-db53acad4694\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") "
Sep 16 21:47:35.960005 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.959961    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-home" (OuterVolumeSpecName: "home") pod "d150203a-6382-4b30-94e1-16131d9f64d5" (UID: "d150203a-6382-4b30-94e1-16131d9f64d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.960005 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.959989    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-model-cache\") pod \"d150203a-6382-4b30-94e1-16131d9f64d5\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") "
Sep 16 21:47:35.960160 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960023    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-home\") pod \"8b303241-58d2-4262-be41-db53acad4694\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") "
Sep 16 21:47:35.960160 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960059    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-tmp-dir\") pod \"01414087-99a7-46bd-ae45-67ecb110c13c\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") "
Sep 16 21:47:35.960160 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960085    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-tmp-dir\") pod \"d150203a-6382-4b30-94e1-16131d9f64d5\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") "
Sep 16 21:47:35.960160 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960117    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d150203a-6382-4b30-94e1-16131d9f64d5-tls-certs\") pod \"d150203a-6382-4b30-94e1-16131d9f64d5\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") "
Sep 16 21:47:35.960160 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960153    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-model-cache\") pod \"01414087-99a7-46bd-ae45-67ecb110c13c\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") "
Sep 16 21:47:35.960429 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960180    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b303241-58d2-4262-be41-db53acad4694-tls-certs\") pod \"8b303241-58d2-4262-be41-db53acad4694\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") "
Sep 16 21:47:35.960429 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960193    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b303241-58d2-4262-be41-db53acad4694" (UID: "8b303241-58d2-4262-be41-db53acad4694"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.960429 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960205    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "d150203a-6382-4b30-94e1-16131d9f64d5" (UID: "d150203a-6382-4b30-94e1-16131d9f64d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.960429 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960214    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k5pc\" (UniqueName: \"kubernetes.io/projected/d150203a-6382-4b30-94e1-16131d9f64d5-kube-api-access-5k5pc\") pod \"d150203a-6382-4b30-94e1-16131d9f64d5\" (UID: \"d150203a-6382-4b30-94e1-16131d9f64d5\") "
Sep 16 21:47:35.960429 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960255    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-dshm\") pod \"8b303241-58d2-4262-be41-db53acad4694\" (UID: \"8b303241-58d2-4262-be41-db53acad4694\") "
Sep 16 21:47:35.960429 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960293    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-dshm\") pod \"01414087-99a7-46bd-ae45-67ecb110c13c\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") "
Sep 16 21:47:35.960429 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960334    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01414087-99a7-46bd-ae45-67ecb110c13c-tls-certs\") pod \"01414087-99a7-46bd-ae45-67ecb110c13c\" (UID: \"01414087-99a7-46bd-ae45-67ecb110c13c\") "
Sep 16 21:47:35.960429 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960346    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d150203a-6382-4b30-94e1-16131d9f64d5" (UID: "d150203a-6382-4b30-94e1-16131d9f64d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.960429 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960373    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-model-cache" (OuterVolumeSpecName: "model-cache") pod "01414087-99a7-46bd-ae45-67ecb110c13c" (UID: "01414087-99a7-46bd-ae45-67ecb110c13c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.960910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960512    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-home" (OuterVolumeSpecName: "home") pod "8b303241-58d2-4262-be41-db53acad4694" (UID: "8b303241-58d2-4262-be41-db53acad4694"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.960910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960721    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-home\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.960910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960740    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-tmp-dir\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.960910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960754    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-home\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.960910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960767    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-model-cache\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.960910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960780    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-model-cache\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.960910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960792    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-home\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.960910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960804    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-tmp-dir\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.960910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960816    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-model-cache\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.961390 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.960980    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01414087-99a7-46bd-ae45-67ecb110c13c" (UID: "01414087-99a7-46bd-ae45-67ecb110c13c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.963415 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.963371    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-dshm" (OuterVolumeSpecName: "dshm") pod "d150203a-6382-4b30-94e1-16131d9f64d5" (UID: "d150203a-6382-4b30-94e1-16131d9f64d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.963544 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.963493    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b303241-58d2-4262-be41-db53acad4694-kube-api-access-vh2vr" (OuterVolumeSpecName: "kube-api-access-vh2vr") pod "8b303241-58d2-4262-be41-db53acad4694" (UID: "8b303241-58d2-4262-be41-db53acad4694"). InnerVolumeSpecName "kube-api-access-vh2vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:35.963654 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.963632    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01414087-99a7-46bd-ae45-67ecb110c13c-kube-api-access-22mqj" (OuterVolumeSpecName: "kube-api-access-22mqj") pod "01414087-99a7-46bd-ae45-67ecb110c13c" (UID: "01414087-99a7-46bd-ae45-67ecb110c13c"). InnerVolumeSpecName "kube-api-access-22mqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:35.963910 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.963884    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d150203a-6382-4b30-94e1-16131d9f64d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d150203a-6382-4b30-94e1-16131d9f64d5" (UID: "d150203a-6382-4b30-94e1-16131d9f64d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:47:35.964011 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.963884    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-dshm" (OuterVolumeSpecName: "dshm") pod "8b303241-58d2-4262-be41-db53acad4694" (UID: "8b303241-58d2-4262-be41-db53acad4694"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.964011 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.963914    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b303241-58d2-4262-be41-db53acad4694-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b303241-58d2-4262-be41-db53acad4694" (UID: "8b303241-58d2-4262-be41-db53acad4694"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:47:35.964011 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.963999    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01414087-99a7-46bd-ae45-67ecb110c13c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01414087-99a7-46bd-ae45-67ecb110c13c" (UID: "01414087-99a7-46bd-ae45-67ecb110c13c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:47:35.964505 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.964486    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d150203a-6382-4b30-94e1-16131d9f64d5-kube-api-access-5k5pc" (OuterVolumeSpecName: "kube-api-access-5k5pc") pod "d150203a-6382-4b30-94e1-16131d9f64d5" (UID: "d150203a-6382-4b30-94e1-16131d9f64d5"). InnerVolumeSpecName "kube-api-access-5k5pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:35.964505 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:35.964489    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-dshm" (OuterVolumeSpecName: "dshm") pod "01414087-99a7-46bd-ae45-67ecb110c13c" (UID: "01414087-99a7-46bd-ae45-67ecb110c13c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:36.062120 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.062090    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh2vr\" (UniqueName: \"kubernetes.io/projected/8b303241-58d2-4262-be41-db53acad4694-kube-api-access-vh2vr\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:36.062120 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.062116    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d150203a-6382-4b30-94e1-16131d9f64d5-dshm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:36.062290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.062127    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22mqj\" (UniqueName: \"kubernetes.io/projected/01414087-99a7-46bd-ae45-67ecb110c13c-kube-api-access-22mqj\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:36.062290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.062137    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-tmp-dir\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:36.062290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.062145    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d150203a-6382-4b30-94e1-16131d9f64d5-tls-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:36.062290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.062153    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b303241-58d2-4262-be41-db53acad4694-tls-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:36.062290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.062160    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k5pc\" (UniqueName: \"kubernetes.io/projected/d150203a-6382-4b30-94e1-16131d9f64d5-kube-api-access-5k5pc\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:36.062290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.062168    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b303241-58d2-4262-be41-db53acad4694-dshm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:36.062290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.062175    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01414087-99a7-46bd-ae45-67ecb110c13c-dshm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:36.062290 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.062182    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01414087-99a7-46bd-ae45-67ecb110c13c-tls-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:47:36.191644 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.191560    2567 generic.go:356] "Generic (PLEG): container finished" podID="d150203a-6382-4b30-94e1-16131d9f64d5" containerID="d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882" exitCode=0
Sep 16 21:47:36.191644 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.191606    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"
Sep 16 21:47:36.191827 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.191641    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd" event={"ID":"d150203a-6382-4b30-94e1-16131d9f64d5","Type":"ContainerDied","Data":"d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882"}
Sep 16 21:47:36.191827 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.191681    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd" event={"ID":"d150203a-6382-4b30-94e1-16131d9f64d5","Type":"ContainerDied","Data":"bf3bc96fb207564d7b21ebd5b10ec80e5424a45e76ce694543229e6bbd31c315"}
Sep 16 21:47:36.191827 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.191705    2567 scope.go:117] "RemoveContainer" containerID="d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882"
Sep 16 21:47:36.193419 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.193207    2567 generic.go:356] "Generic (PLEG): container finished" podID="8b303241-58d2-4262-be41-db53acad4694" containerID="dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb" exitCode=0
Sep 16 21:47:36.193419 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.193240    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz" event={"ID":"8b303241-58d2-4262-be41-db53acad4694","Type":"ContainerDied","Data":"dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb"}
Sep 16 21:47:36.193419 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.193260    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"
Sep 16 21:47:36.193419 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.193278    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz" event={"ID":"8b303241-58d2-4262-be41-db53acad4694","Type":"ContainerDied","Data":"2bcbbde3ea86071086e515cf0d0a64f0d100887a567a650df04307f3211a3ea8"}
Sep 16 21:47:36.194825 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.194800    2567 generic.go:356] "Generic (PLEG): container finished" podID="01414087-99a7-46bd-ae45-67ecb110c13c" containerID="4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393" exitCode=0
Sep 16 21:47:36.194947 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.194850    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"
Sep 16 21:47:36.194947 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.194875    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h" event={"ID":"01414087-99a7-46bd-ae45-67ecb110c13c","Type":"ContainerDied","Data":"4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393"}
Sep 16 21:47:36.194947 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.194903    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h" event={"ID":"01414087-99a7-46bd-ae45-67ecb110c13c","Type":"ContainerDied","Data":"23d4d0cf6ce3b0d939f641da54d5b6f29c94882e6955250a715865f559ea632c"}
Sep 16 21:47:36.202386 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.202364    2567 scope.go:117] "RemoveContainer" containerID="d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882"
Sep 16 21:47:36.202720 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:47:36.202703    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882\": container with ID starting with d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882 not found: ID does not exist" containerID="d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882"
Sep 16 21:47:36.202790 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.202729    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882"} err="failed to get container status \"d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882\": rpc error: code = NotFound desc = could not find container \"d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882\": container with ID starting with d514884fe04cc2f25d35dce52416674b5f99a2af906e849b43185a3cc1d06882 not found: ID does not exist"
Sep 16 21:47:36.202790 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.202744    2567 scope.go:117] "RemoveContainer" containerID="dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb"
Sep 16 21:47:36.211508 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.211490    2567 scope.go:117] "RemoveContainer" containerID="dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb"
Sep 16 21:47:36.211739 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:47:36.211722    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb\": container with ID starting with dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb not found: ID does not exist" containerID="dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb"
Sep 16 21:47:36.211799 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.211747    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb"} err="failed to get container status \"dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb\": rpc error: code = NotFound desc = could not find container \"dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb\": container with ID starting with dca94409a290126da6a7bd1d922b0d363b6815a8306febb23d9d0c29f5db5feb not found: ID does not exist"
Sep 16 21:47:36.211799 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.211763    2567 scope.go:117] "RemoveContainer" containerID="4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393"
Sep 16 21:47:36.217582 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.217562    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"]
Sep 16 21:47:36.220168 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.220084    2567 scope.go:117] "RemoveContainer" containerID="4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393"
Sep 16 21:47:36.220829 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:47:36.220809    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393\": container with ID starting with 4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393 not found: ID does not exist" containerID="4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393"
Sep 16 21:47:36.220914 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.220838    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393"} err="failed to get container status \"4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393\": rpc error: code = NotFound desc = could not find container \"4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393\": container with ID starting with 4aef6d7df5f7fa6e1a6ab128a86e8a5cfde59baa301833e360fdddf69dad1393 not found: ID does not exist"
Sep 16 21:47:36.223188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.223167    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-2-w5-e047d8-kserve-7557cf8b89-hcgmd"]
Sep 16 21:47:36.232580 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.232560    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"]
Sep 16 21:47:36.235855 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.235830    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/distinct-w5-e047d8-kserve-64fc84f546-s7t2h"]
Sep 16 21:47:36.246173 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.246153    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"]
Sep 16 21:47:36.249747 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:36.249727    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/premium-w5-e047d8-kserve-56d6656c7f-x5bbz"]
Sep 16 21:47:37.108925 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:37.108891    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01414087-99a7-46bd-ae45-67ecb110c13c" path="/var/lib/kubelet/pods/01414087-99a7-46bd-ae45-67ecb110c13c/volumes"
Sep 16 21:47:37.109291 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:37.109275    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b303241-58d2-4262-be41-db53acad4694" path="/var/lib/kubelet/pods/8b303241-58d2-4262-be41-db53acad4694/volumes"
Sep 16 21:47:37.109670 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:37.109656    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d150203a-6382-4b30-94e1-16131d9f64d5" path="/var/lib/kubelet/pods/d150203a-6382-4b30-94e1-16131d9f64d5/volumes"
Sep 16 21:47:51.786042 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:51.786006    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:47:52.969577 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:47:52.969537    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"]
Sep 16 21:48:01.581022 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:01.580986    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:48:07.988763 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:07.988657    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9" podUID="5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" containerName="main" containerID="cri-o://9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce" gracePeriod=45
Sep 16 21:48:08.234330 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.234303    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:48:08.342601 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.342529    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-model-cache\") pod \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") "
Sep 16 21:48:08.342601 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.342570    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-home\") pod \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") "
Sep 16 21:48:08.342601 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.342591    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tls-certs\") pod \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") "
Sep 16 21:48:08.342882 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.342607    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drv95\" (UniqueName: \"kubernetes.io/projected/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-kube-api-access-drv95\") pod \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") "
Sep 16 21:48:08.342882 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.342646    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-dshm\") pod \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") "
Sep 16 21:48:08.342882 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.342668    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tmp-dir\") pod \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\" (UID: \"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f\") "
Sep 16 21:48:08.342882 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.342855    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" (UID: "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:48:08.343055 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.342909    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-home" (OuterVolumeSpecName: "home") pod "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" (UID: "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:48:08.343055 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.342982    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" (UID: "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:48:08.344847 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.344822    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-dshm" (OuterVolumeSpecName: "dshm") pod "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" (UID: "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:48:08.344847 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.344834    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" (UID: "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:48:08.345014 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.344878    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-kube-api-access-drv95" (OuterVolumeSpecName: "kube-api-access-drv95") pod "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" (UID: "5b6d29b2-8aec-4bf5-a69b-24fecc57a19f"). InnerVolumeSpecName "kube-api-access-drv95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:48:08.390600 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.390571    2567 generic.go:356] "Generic (PLEG): container finished" podID="5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" containerID="9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce" exitCode=0
Sep 16 21:48:08.390716 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.390631    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"
Sep 16 21:48:08.390716 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.390655    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9" event={"ID":"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f","Type":"ContainerDied","Data":"9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce"}
Sep 16 21:48:08.390716 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.390696    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9" event={"ID":"5b6d29b2-8aec-4bf5-a69b-24fecc57a19f","Type":"ContainerDied","Data":"9f729ef3b6ad36194861714eb1e53ef70169659ad217acf46292cdcf0a613c18"}
Sep 16 21:48:08.390716 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.390712    2567 scope.go:117] "RemoveContainer" containerID="9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce"
Sep 16 21:48:08.400379 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.400364    2567 scope.go:117] "RemoveContainer" containerID="9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce"
Sep 16 21:48:08.400657 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:48:08.400641    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce\": container with ID starting with 9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce not found: ID does not exist" containerID="9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce"
Sep 16 21:48:08.400703 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.400666    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce"} err="failed to get container status \"9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce\": rpc error: code = NotFound desc = could not find container \"9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce\": container with ID starting with 9b4d975bb709fdf3c4f806c82f4b9376b0e23649df2301e529240969295d42ce not found: ID does not exist"
Sep 16 21:48:08.411645 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.411622    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"]
Sep 16 21:48:08.413726 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.413708    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/auth-test-model-1b68db-kserve-7bb7c558d7-8wwk9"]
Sep 16 21:48:08.443563 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.443539    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-dshm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:48:08.443636 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.443578    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tmp-dir\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:48:08.443636 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.443588    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-model-cache\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:48:08.443636 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.443602    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-home\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:48:08.443636 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.443611    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-tls-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:48:08.443636 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:08.443619    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drv95\" (UniqueName: \"kubernetes.io/projected/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f-kube-api-access-drv95\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:48:09.109049 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:09.109017    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" path="/var/lib/kubelet/pods/5b6d29b2-8aec-4bf5-a69b-24fecc57a19f/volumes"
Sep 16 21:48:26.189780 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:26.189742    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:48:48.081233 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:48:48.081189    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:49:03.375337 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:03.375302    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:49:16.280115 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:16.280077    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:49:24.587464 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.587390    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"]
Sep 16 21:49:24.588068 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588035    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa" containerName="payload-pre-processing"
Sep 16 21:49:24.588068 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588067    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa" containerName="payload-pre-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588078    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c" containerName="payload-pre-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588084    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c" containerName="payload-pre-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588094    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d150203a-6382-4b30-94e1-16131d9f64d5" containerName="main"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588100    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d150203a-6382-4b30-94e1-16131d9f64d5" containerName="main"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588108    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048635a3-c291-4287-8e3b-0fc3d0539688" containerName="payload-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588114    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="048635a3-c291-4287-8e3b-0fc3d0539688" containerName="payload-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588122    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" containerName="main"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588127    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" containerName="main"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588134    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78fcf47-01b0-4506-9115-5ec7eefaf5e1" containerName="payload-pre-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588139    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78fcf47-01b0-4506-9115-5ec7eefaf5e1" containerName="payload-pre-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588150    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725c29ad-36fa-4062-b369-4e2ee78500d7" containerName="payload-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588156    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="725c29ad-36fa-4062-b369-4e2ee78500d7" containerName="payload-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588165    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="466e6ec5-8ed5-47d4-a566-a009a0fadc4c" containerName="payload-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588170    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="466e6ec5-8ed5-47d4-a566-a009a0fadc4c" containerName="payload-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588178    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1737eb4-8352-4cad-ac0a-651aecc1ad02" containerName="payload-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588183    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1737eb4-8352-4cad-ac0a-651aecc1ad02" containerName="payload-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588195    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37fadcd3-04b4-4594-8af3-06abbd57efc8" containerName="payload-pre-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588200    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="37fadcd3-04b4-4594-8af3-06abbd57efc8" containerName="payload-pre-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588205    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de1b7dcd-d05f-4449-9618-1bd9f23a6e0a" containerName="payload-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588210    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="de1b7dcd-d05f-4449-9618-1bd9f23a6e0a" containerName="payload-processing"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588218    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01414087-99a7-46bd-ae45-67ecb110c13c" containerName="main"
Sep 16 21:49:24.588217 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588223    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="01414087-99a7-46bd-ae45-67ecb110c13c" containerName="main"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588230    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b303241-58d2-4262-be41-db53acad4694" containerName="main"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588235    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b303241-58d2-4262-be41-db53acad4694" containerName="main"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588243    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f" containerName="payload-pre-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588250    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f" containerName="payload-pre-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588255    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deabc723-92ef-472c-a5ae-2a9a3678e012" containerName="payload-pre-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588260    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="deabc723-92ef-472c-a5ae-2a9a3678e012" containerName="payload-pre-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588324    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="de1b7dcd-d05f-4449-9618-1bd9f23a6e0a" containerName="payload-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588332    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="048635a3-c291-4287-8e3b-0fc3d0539688" containerName="payload-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588339    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b303241-58d2-4262-be41-db53acad4694" containerName="main"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588345    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="725c29ad-36fa-4062-b369-4e2ee78500d7" containerName="payload-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588350    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="deabc723-92ef-472c-a5ae-2a9a3678e012" containerName="payload-pre-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588355    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="01414087-99a7-46bd-ae45-67ecb110c13c" containerName="main"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588359    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d150203a-6382-4b30-94e1-16131d9f64d5" containerName="main"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588365    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="466e6ec5-8ed5-47d4-a566-a009a0fadc4c" containerName="payload-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588372    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="37fadcd3-04b4-4594-8af3-06abbd57efc8" containerName="payload-pre-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588376    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c78fcf47-01b0-4506-9115-5ec7eefaf5e1" containerName="payload-pre-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588381    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6cfca04-4a9c-4aa3-8d8e-0a35bd53405c" containerName="payload-pre-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588387    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ecd51e-7c26-4d3f-83ae-ddb9c4f5dc6f" containerName="payload-pre-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588391    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b6d29b2-8aec-4bf5-a69b-24fecc57a19f" containerName="main"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588418    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1737eb4-8352-4cad-ac0a-651aecc1ad02" containerName="payload-processing"
Sep 16 21:49:24.588916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.588423    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b5ef76-65e6-4f9e-a7ee-d4b079644bfa" containerName="payload-pre-processing"
Sep 16 21:49:24.591929 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.591912    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.594177 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.594154    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"kube-root-ca.crt\""
Sep 16 21:49:24.594840 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.594815    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"default-dockercfg-vvzqj\""
Sep 16 21:49:24.594937 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.594825    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"openshift-service-ca.crt\""
Sep 16 21:49:24.595140 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.595124    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"rate-test-model-a7df81-kserve-self-signed-certs\""
Sep 16 21:49:24.600448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.600425    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"]
Sep 16 21:49:24.683374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.683339    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgxd\" (UniqueName: \"kubernetes.io/projected/ef90ba4d-bd82-4335-9376-d5736437532e-kube-api-access-sqgxd\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.683548 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.683389    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-model-cache\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.683548 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.683504    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef90ba4d-bd82-4335-9376-d5736437532e-tls-certs\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.683625 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.683571    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-tmp-dir\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.683625 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.683597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-home\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.683693 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.683627    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-dshm\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.785075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.785040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-dshm\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.785260 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.785083    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgxd\" (UniqueName: \"kubernetes.io/projected/ef90ba4d-bd82-4335-9376-d5736437532e-kube-api-access-sqgxd\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.785260 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.785108    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-model-cache\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.785260 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.785149    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef90ba4d-bd82-4335-9376-d5736437532e-tls-certs\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.785260 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.785191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-tmp-dir\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.785260 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.785222    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-home\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.785590 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.785568    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-model-cache\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.785630 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.785611    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-tmp-dir\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.785666 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.785637    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-home\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.787366 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.787347    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-dshm\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.787563 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.787544    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef90ba4d-bd82-4335-9376-d5736437532e-tls-certs\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.792346 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.792327    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgxd\" (UniqueName: \"kubernetes.io/projected/ef90ba4d-bd82-4335-9376-d5736437532e-kube-api-access-sqgxd\") pod \"rate-test-model-a7df81-kserve-848d886f5-726zh\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:24.901578 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:24.901505    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:25.174091 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:25.174064    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"]
Sep 16 21:49:25.175890 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:49:25.175862    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef90ba4d_bd82_4335_9376_d5736437532e.slice/crio-cc6b03128e1789a68ec8af08c665baa77886983f6e4ed1183c83695b648a5719 WatchSource:0}: Error finding container cc6b03128e1789a68ec8af08c665baa77886983f6e4ed1183c83695b648a5719: Status 404 returned error can't find the container with id cc6b03128e1789a68ec8af08c665baa77886983f6e4ed1183c83695b648a5719
Sep 16 21:49:25.844487 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:25.844450    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh" event={"ID":"ef90ba4d-bd82-4335-9376-d5736437532e","Type":"ContainerStarted","Data":"02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72"}
Sep 16 21:49:25.844866 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:25.844503    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh" event={"ID":"ef90ba4d-bd82-4335-9376-d5736437532e","Type":"ContainerStarted","Data":"cc6b03128e1789a68ec8af08c665baa77886983f6e4ed1183c83695b648a5719"}
Sep 16 21:49:25.860954 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:25.860905    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh" podStartSLOduration=1.860889349 podStartE2EDuration="1.860889349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:49:25.858758647 +0000 UTC m=+1605.244441761" watchObservedRunningTime="2026-09-16 21:49:25.860889349 +0000 UTC m=+1605.246572432"
Sep 16 21:49:26.773483 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:26.773440    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:49:34.902456 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:34.902348    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:34.902456 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:34.902428    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:34.915101 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:34.915073    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:35.924528 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:35.924493    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:49:41.455774 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:49:41.455744    2567 scope.go:117] "RemoveContainer" containerID="4bb81e7744749d60e2e90e372d4e0b9fddef26b0df79521bc11978df3ed37617"
Sep 16 21:50:13.042133 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:50:13.042098    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:50:51.076425 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:50:51.076373    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:51:00.178442 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:00.178386    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:51:04.938503 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:04.938421    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"]
Sep 16 21:51:10.383463 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:10.383392    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:51:11.819983 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.819951    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"]
Sep 16 21:51:11.823879 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.823854    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:51:11.825887 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.825853    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ad0222\""
Sep 16 21:51:11.826020 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.825896    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ad0222-dockercfg-4jv9d\""
Sep 16 21:51:11.835063 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.835036    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"]
Sep 16 21:51:11.843626 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.843603    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"]
Sep 16 21:51:11.846921 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.846905    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:51:11.859157 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.859129    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"]
Sep 16 21:51:11.875156 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.875116    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2\" (UID: \"e626ed97-e7b3-402c-a6cf-ad17d0bbd344\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:51:11.875315 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.875176    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6192bb3-09bf-4341-8520-55e0b2fb0170-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt\" (UID: \"b6192bb3-09bf-4341-8520-55e0b2fb0170\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:51:11.875315 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.875285    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkfmf\" (UniqueName: \"kubernetes.io/projected/b6192bb3-09bf-4341-8520-55e0b2fb0170-kube-api-access-tkfmf\") pod \"payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt\" (UID: \"b6192bb3-09bf-4341-8520-55e0b2fb0170\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:51:11.875448 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.875337    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2t27\" (UniqueName: \"kubernetes.io/projected/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-kube-api-access-k2t27\") pod \"payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2\" (UID: \"e626ed97-e7b3-402c-a6cf-ad17d0bbd344\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:51:11.975917 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.975884    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2\" (UID: \"e626ed97-e7b3-402c-a6cf-ad17d0bbd344\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:51:11.976092 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.975924    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6192bb3-09bf-4341-8520-55e0b2fb0170-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt\" (UID: \"b6192bb3-09bf-4341-8520-55e0b2fb0170\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:51:11.976092 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.975949    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkfmf\" (UniqueName: \"kubernetes.io/projected/b6192bb3-09bf-4341-8520-55e0b2fb0170-kube-api-access-tkfmf\") pod \"payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt\" (UID: \"b6192bb3-09bf-4341-8520-55e0b2fb0170\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:51:11.976092 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.976006    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2t27\" (UniqueName: \"kubernetes.io/projected/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-kube-api-access-k2t27\") pod \"payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2\" (UID: \"e626ed97-e7b3-402c-a6cf-ad17d0bbd344\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:51:11.976709 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.976692    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2\" (UID: \"e626ed97-e7b3-402c-a6cf-ad17d0bbd344\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:51:11.976762 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.976725    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6192bb3-09bf-4341-8520-55e0b2fb0170-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt\" (UID: \"b6192bb3-09bf-4341-8520-55e0b2fb0170\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:51:11.983189 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.983158    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2t27\" (UniqueName: \"kubernetes.io/projected/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-kube-api-access-k2t27\") pod \"payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2\" (UID: \"e626ed97-e7b3-402c-a6cf-ad17d0bbd344\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:51:11.983471 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:11.983450    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkfmf\" (UniqueName: \"kubernetes.io/projected/b6192bb3-09bf-4341-8520-55e0b2fb0170-kube-api-access-tkfmf\") pod \"payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt\" (UID: \"b6192bb3-09bf-4341-8520-55e0b2fb0170\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:51:12.137247 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:12.137166    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:51:12.156999 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:12.156972    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:51:12.408829 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:12.408605    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"]
Sep 16 21:51:12.411118 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:51:12.411094    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6192bb3_09bf_4341_8520_55e0b2fb0170.slice/crio-06e5ec048730295521909f626d4b44adc7dc7cc7079e7489911e6f8b9a5f5111 WatchSource:0}: Error finding container 06e5ec048730295521909f626d4b44adc7dc7cc7079e7489911e6f8b9a5f5111: Status 404 returned error can't find the container with id 06e5ec048730295521909f626d4b44adc7dc7cc7079e7489911e6f8b9a5f5111
Sep 16 21:51:12.439734 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:12.439712    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"]
Sep 16 21:51:12.447505 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:51:12.447475    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode626ed97_e7b3_402c_a6cf_ad17d0bbd344.slice/crio-5abf992e03547d766ac3a65a68c1e0635aa89cab4c1e771613331f05ef53be44 WatchSource:0}: Error finding container 5abf992e03547d766ac3a65a68c1e0635aa89cab4c1e771613331f05ef53be44: Status 404 returned error can't find the container with id 5abf992e03547d766ac3a65a68c1e0635aa89cab4c1e771613331f05ef53be44
Sep 16 21:51:12.460668 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:12.460638    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt" event={"ID":"b6192bb3-09bf-4341-8520-55e0b2fb0170","Type":"ContainerStarted","Data":"06e5ec048730295521909f626d4b44adc7dc7cc7079e7489911e6f8b9a5f5111"}
Sep 16 21:51:12.461803 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:12.461779    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2" event={"ID":"e626ed97-e7b3-402c-a6cf-ad17d0bbd344","Type":"ContainerStarted","Data":"5abf992e03547d766ac3a65a68c1e0635aa89cab4c1e771613331f05ef53be44"}
Sep 16 21:51:13.469628 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:13.469592    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt" event={"ID":"b6192bb3-09bf-4341-8520-55e0b2fb0170","Type":"ContainerStarted","Data":"743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f"}
Sep 16 21:51:13.470098 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:13.469693    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:51:13.471391 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:13.471359    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2" event={"ID":"e626ed97-e7b3-402c-a6cf-ad17d0bbd344","Type":"ContainerStarted","Data":"c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094"}
Sep 16 21:51:13.471534 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:13.471431    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:51:13.485613 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:13.485570    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt" podStartSLOduration=2.485556429 podStartE2EDuration="2.485556429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:51:13.483712562 +0000 UTC m=+1712.869395686" watchObservedRunningTime="2026-09-16 21:51:13.485556429 +0000 UTC m=+1712.871239512"
Sep 16 21:51:13.499456 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:13.499418    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2" podStartSLOduration=2.4993914569999998 podStartE2EDuration="2.499391457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:51:13.497566533 +0000 UTC m=+1712.883249648" watchObservedRunningTime="2026-09-16 21:51:13.499391457 +0000 UTC m=+1712.885074540"
Sep 16 21:51:19.957759 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:19.957720    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh" podUID="ef90ba4d-bd82-4335-9376-d5736437532e" containerName="main" containerID="cri-o://02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72" gracePeriod=45
Sep 16 21:51:20.190211 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.190185    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:51:20.247516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247446    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-tmp-dir\") pod \"ef90ba4d-bd82-4335-9376-d5736437532e\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") "
Sep 16 21:51:20.247516 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247498    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqgxd\" (UniqueName: \"kubernetes.io/projected/ef90ba4d-bd82-4335-9376-d5736437532e-kube-api-access-sqgxd\") pod \"ef90ba4d-bd82-4335-9376-d5736437532e\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") "
Sep 16 21:51:20.247736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247534    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-home\") pod \"ef90ba4d-bd82-4335-9376-d5736437532e\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") "
Sep 16 21:51:20.247736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247566    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-model-cache\") pod \"ef90ba4d-bd82-4335-9376-d5736437532e\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") "
Sep 16 21:51:20.247736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247600    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef90ba4d-bd82-4335-9376-d5736437532e-tls-certs\") pod \"ef90ba4d-bd82-4335-9376-d5736437532e\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") "
Sep 16 21:51:20.247736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247631    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-dshm\") pod \"ef90ba4d-bd82-4335-9376-d5736437532e\" (UID: \"ef90ba4d-bd82-4335-9376-d5736437532e\") "
Sep 16 21:51:20.247736 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247720    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef90ba4d-bd82-4335-9376-d5736437532e" (UID: "ef90ba4d-bd82-4335-9376-d5736437532e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:51:20.247963 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247837    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef90ba4d-bd82-4335-9376-d5736437532e" (UID: "ef90ba4d-bd82-4335-9376-d5736437532e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:51:20.247963 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247848    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-home" (OuterVolumeSpecName: "home") pod "ef90ba4d-bd82-4335-9376-d5736437532e" (UID: "ef90ba4d-bd82-4335-9376-d5736437532e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:51:20.247963 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247955    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-tmp-dir\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:51:20.248130 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247973    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-home\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:51:20.248130 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.247982    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-model-cache\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:51:20.249887 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.249858    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef90ba4d-bd82-4335-9376-d5736437532e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef90ba4d-bd82-4335-9376-d5736437532e" (UID: "ef90ba4d-bd82-4335-9376-d5736437532e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:51:20.249887 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.249874    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef90ba4d-bd82-4335-9376-d5736437532e-kube-api-access-sqgxd" (OuterVolumeSpecName: "kube-api-access-sqgxd") pod "ef90ba4d-bd82-4335-9376-d5736437532e" (UID: "ef90ba4d-bd82-4335-9376-d5736437532e"). InnerVolumeSpecName "kube-api-access-sqgxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:51:20.250029 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.249917    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-dshm" (OuterVolumeSpecName: "dshm") pod "ef90ba4d-bd82-4335-9376-d5736437532e" (UID: "ef90ba4d-bd82-4335-9376-d5736437532e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:51:20.349335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.349300    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqgxd\" (UniqueName: \"kubernetes.io/projected/ef90ba4d-bd82-4335-9376-d5736437532e-kube-api-access-sqgxd\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:51:20.349335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.349329    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef90ba4d-bd82-4335-9376-d5736437532e-tls-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:51:20.349335 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.349339    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef90ba4d-bd82-4335-9376-d5736437532e-dshm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:51:20.521973 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.521869    2567 generic.go:356] "Generic (PLEG): container finished" podID="ef90ba4d-bd82-4335-9376-d5736437532e" containerID="02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72" exitCode=0
Sep 16 21:51:20.521973 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.521936    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"
Sep 16 21:51:20.522188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.521969    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh" event={"ID":"ef90ba4d-bd82-4335-9376-d5736437532e","Type":"ContainerDied","Data":"02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72"}
Sep 16 21:51:20.522188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.522024    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh" event={"ID":"ef90ba4d-bd82-4335-9376-d5736437532e","Type":"ContainerDied","Data":"cc6b03128e1789a68ec8af08c665baa77886983f6e4ed1183c83695b648a5719"}
Sep 16 21:51:20.522188 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.522044    2567 scope.go:117] "RemoveContainer" containerID="02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72"
Sep 16 21:51:20.532705 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.532685    2567 scope.go:117] "RemoveContainer" containerID="02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72"
Sep 16 21:51:20.532977 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:51:20.532957    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72\": container with ID starting with 02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72 not found: ID does not exist" containerID="02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72"
Sep 16 21:51:20.533046 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.532988    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72"} err="failed to get container status \"02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72\": rpc error: code = NotFound desc = could not find container \"02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72\": container with ID starting with 02d24a8b03c7e96d1d6eb22cd0fad6b9c45e9c9d61add6cb3c0f1b40b5a56a72 not found: ID does not exist"
Sep 16 21:51:20.544547 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.544521    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"]
Sep 16 21:51:20.547873 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.547847    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/rate-test-model-a7df81-kserve-848d886f5-726zh"]
Sep 16 21:51:20.894098 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:20.892491    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:51:21.108750 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:21.108717    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef90ba4d-bd82-4335-9376-d5736437532e" path="/var/lib/kubelet/pods/ef90ba4d-bd82-4335-9376-d5736437532e/volumes"
Sep 16 21:51:24.480304 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:24.480274    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:51:24.480816 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:24.480339    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:51:41.489011 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:41.488976    2567 scope.go:117] "RemoveContainer" containerID="44da05de4980dbe74b6fc32f3869e3b2b17819def997a751ecb2e94f9968fb64"
Sep 16 21:51:41.497831 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:41.497802    2567 scope.go:117] "RemoveContainer" containerID="2ae0f022424901f4f449206d4f418924d1938ac2980201b2ab6c931a5c19ea66"
Sep 16 21:51:42.326615 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.326577    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"]
Sep 16 21:51:42.327023 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.327011    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef90ba4d-bd82-4335-9376-d5736437532e" containerName="main"
Sep 16 21:51:42.327077 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.327025    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef90ba4d-bd82-4335-9376-d5736437532e" containerName="main"
Sep 16 21:51:42.327115 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.327095    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef90ba4d-bd82-4335-9376-d5736437532e" containerName="main"
Sep 16 21:51:42.332075 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.332057    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:51:42.334340 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.334320    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-03eca6\""
Sep 16 21:51:42.334475 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.334456    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-03eca6-dockercfg-dvl77\""
Sep 16 21:51:42.341215 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.341192    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"]
Sep 16 21:51:42.346872 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.346841    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"]
Sep 16 21:51:42.350574 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.350557    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:51:42.365165 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.365142    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"]
Sep 16 21:51:42.438295 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.438265    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2st\" (UniqueName: \"kubernetes.io/projected/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-kube-api-access-pg2st\") pod \"payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9\" (UID: \"8d14c8e0-41b7-4bfd-905b-8988dd0555c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:51:42.438482 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.438306    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e01938-b669-467e-9f3e-5489019b56fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49\" (UID: \"21e01938-b669-467e-9f3e-5489019b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:51:42.438482 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.438352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6n7\" (UniqueName: \"kubernetes.io/projected/21e01938-b669-467e-9f3e-5489019b56fd-kube-api-access-bx6n7\") pod \"payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49\" (UID: \"21e01938-b669-467e-9f3e-5489019b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:51:42.438482 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.438380    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9\" (UID: \"8d14c8e0-41b7-4bfd-905b-8988dd0555c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:51:42.539721 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.539682    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2st\" (UniqueName: \"kubernetes.io/projected/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-kube-api-access-pg2st\") pod \"payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9\" (UID: \"8d14c8e0-41b7-4bfd-905b-8988dd0555c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:51:42.540147 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.539728    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e01938-b669-467e-9f3e-5489019b56fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49\" (UID: \"21e01938-b669-467e-9f3e-5489019b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:51:42.540147 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.539781    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6n7\" (UniqueName: \"kubernetes.io/projected/21e01938-b669-467e-9f3e-5489019b56fd-kube-api-access-bx6n7\") pod \"payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49\" (UID: \"21e01938-b669-467e-9f3e-5489019b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:51:42.540147 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.539809    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9\" (UID: \"8d14c8e0-41b7-4bfd-905b-8988dd0555c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:51:42.540616 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.540596    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9\" (UID: \"8d14c8e0-41b7-4bfd-905b-8988dd0555c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:51:42.540755 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.540729    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e01938-b669-467e-9f3e-5489019b56fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49\" (UID: \"21e01938-b669-467e-9f3e-5489019b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:51:42.547422 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.547377    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6n7\" (UniqueName: \"kubernetes.io/projected/21e01938-b669-467e-9f3e-5489019b56fd-kube-api-access-bx6n7\") pod \"payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49\" (UID: \"21e01938-b669-467e-9f3e-5489019b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:51:42.547537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.547482    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2st\" (UniqueName: \"kubernetes.io/projected/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-kube-api-access-pg2st\") pod \"payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9\" (UID: \"8d14c8e0-41b7-4bfd-905b-8988dd0555c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:51:42.644131 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.644065    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:51:42.661445 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.661411    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:51:42.927767 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.927734    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"]
Sep 16 21:51:42.929683 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:51:42.929652    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e01938_b669_467e_9f3e_5489019b56fd.slice/crio-991334a4569f841149abe26924d9615f92aa72b5e9642843498c3dd1d10cbfb0 WatchSource:0}: Error finding container 991334a4569f841149abe26924d9615f92aa72b5e9642843498c3dd1d10cbfb0: Status 404 returned error can't find the container with id 991334a4569f841149abe26924d9615f92aa72b5e9642843498c3dd1d10cbfb0
Sep 16 21:51:42.955311 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:42.955288    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"]
Sep 16 21:51:42.957450 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:51:42.957389    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d14c8e0_41b7_4bfd_905b_8988dd0555c3.slice/crio-7580c40a3b46f7b8b324ee10497297febf3fa043f5a59908c61318e636506648 WatchSource:0}: Error finding container 7580c40a3b46f7b8b324ee10497297febf3fa043f5a59908c61318e636506648: Status 404 returned error can't find the container with id 7580c40a3b46f7b8b324ee10497297febf3fa043f5a59908c61318e636506648
Sep 16 21:51:43.664612 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:43.664575    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49" event={"ID":"21e01938-b669-467e-9f3e-5489019b56fd","Type":"ContainerStarted","Data":"a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2"}
Sep 16 21:51:43.665059 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:43.664629    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49" event={"ID":"21e01938-b669-467e-9f3e-5489019b56fd","Type":"ContainerStarted","Data":"991334a4569f841149abe26924d9615f92aa72b5e9642843498c3dd1d10cbfb0"}
Sep 16 21:51:43.665059 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:43.664758    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:51:43.666343 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:43.666313    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9" event={"ID":"8d14c8e0-41b7-4bfd-905b-8988dd0555c3","Type":"ContainerStarted","Data":"85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c"}
Sep 16 21:51:43.666472 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:43.666368    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9" event={"ID":"8d14c8e0-41b7-4bfd-905b-8988dd0555c3","Type":"ContainerStarted","Data":"7580c40a3b46f7b8b324ee10497297febf3fa043f5a59908c61318e636506648"}
Sep 16 21:51:43.666531 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:43.666484    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:51:43.683133 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:43.683093    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49" podStartSLOduration=1.6830793480000001 podStartE2EDuration="1.683079348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:51:43.68072603 +0000 UTC m=+1743.066409141" watchObservedRunningTime="2026-09-16 21:51:43.683079348 +0000 UTC m=+1743.068762431"
Sep 16 21:51:43.700990 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:43.700945    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9" podStartSLOduration=1.7009271419999998 podStartE2EDuration="1.700927142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:51:43.698039004 +0000 UTC m=+1743.083722089" watchObservedRunningTime="2026-09-16 21:51:43.700927142 +0000 UTC m=+1743.086610227"
Sep 16 21:51:46.793277 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:46.793237    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:51:54.673454 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:54.673414    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:51:54.673866 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:51:54.673503    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:52:08.802006 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:08.801975    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-px5dq_2babf7f2-b5ca-4aa7-af2a-bd60178bac8c/payload-processing/0.log"
Sep 16 21:52:08.910647 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:08.910619    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9_8d14c8e0-41b7-4bfd-905b-8988dd0555c3/payload-processing/0.log"
Sep 16 21:52:26.193960 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:26.193923    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:52:39.282961 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:39.282882    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:52:44.854882 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:44.854831    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2_e626ed97-e7b3-402c-a6cf-ad17d0bbd344/payload-processing/0.log"
Sep 16 21:52:44.974124 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:44.974082    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-px5dq_2babf7f2-b5ca-4aa7-af2a-bd60178bac8c/payload-processing/0.log"
Sep 16 21:52:45.087110 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:45.087080    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9_8d14c8e0-41b7-4bfd-905b-8988dd0555c3/payload-processing/0.log"
Sep 16 21:52:45.198914 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:45.198886    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9_8d14c8e0-41b7-4bfd-905b-8988dd0555c3/payload-processing/0.log"
Sep 16 21:52:53.058110 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.058071    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"]
Sep 16 21:52:53.061708 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.061688    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:52:53.064050 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.064029    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-dcbfbc\""
Sep 16 21:52:53.064236 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.064213    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-dcbfbc-dockercfg-68sbd\""
Sep 16 21:52:53.072726 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.072704    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"]
Sep 16 21:52:53.098332 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.098306    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"]
Sep 16 21:52:53.101890 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.101870    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:52:53.113782 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.113755    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"]
Sep 16 21:52:53.165437 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.165388    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8kck\" (UniqueName: \"kubernetes.io/projected/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-kube-api-access-v8kck\") pod \"payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh\" (UID: \"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:52:53.165591 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.165459    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh\" (UID: \"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:52:53.165591 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.165486    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjfhm\" (UniqueName: \"kubernetes.io/projected/c2430b5e-0754-405e-afdf-571515bfcee5-kube-api-access-cjfhm\") pod \"payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2\" (UID: \"c2430b5e-0754-405e-afdf-571515bfcee5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:52:53.165591 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.165514    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2430b5e-0754-405e-afdf-571515bfcee5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2\" (UID: \"c2430b5e-0754-405e-afdf-571515bfcee5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:52:53.266779 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.266739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kck\" (UniqueName: \"kubernetes.io/projected/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-kube-api-access-v8kck\") pod \"payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh\" (UID: \"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:52:53.266779 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.266787    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh\" (UID: \"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:52:53.267038 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.266807    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjfhm\" (UniqueName: \"kubernetes.io/projected/c2430b5e-0754-405e-afdf-571515bfcee5-kube-api-access-cjfhm\") pod \"payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2\" (UID: \"c2430b5e-0754-405e-afdf-571515bfcee5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:52:53.267038 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.266834    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2430b5e-0754-405e-afdf-571515bfcee5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2\" (UID: \"c2430b5e-0754-405e-afdf-571515bfcee5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:52:53.267510 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.267488    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh\" (UID: \"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:52:53.267574 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.267522    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2430b5e-0754-405e-afdf-571515bfcee5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2\" (UID: \"c2430b5e-0754-405e-afdf-571515bfcee5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:52:53.274696 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.274675    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kck\" (UniqueName: \"kubernetes.io/projected/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-kube-api-access-v8kck\") pod \"payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh\" (UID: \"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:52:53.274795 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.274680    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjfhm\" (UniqueName: \"kubernetes.io/projected/c2430b5e-0754-405e-afdf-571515bfcee5-kube-api-access-cjfhm\") pod \"payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2\" (UID: \"c2430b5e-0754-405e-afdf-571515bfcee5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:52:53.373457 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.373333    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:52:53.415223 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.415191    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:52:53.653941 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.653910    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"]
Sep 16 21:52:53.655751 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:52:53.655723    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2430b5e_0754_405e_afdf_571515bfcee5.slice/crio-4b519fda43a4d6d935ef3167fdc78f6e01dfd0243669a14837f62a51e588a8ae WatchSource:0}: Error finding container 4b519fda43a4d6d935ef3167fdc78f6e01dfd0243669a14837f62a51e588a8ae: Status 404 returned error can't find the container with id 4b519fda43a4d6d935ef3167fdc78f6e01dfd0243669a14837f62a51e588a8ae
Sep 16 21:52:53.702274 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:53.702246    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"]
Sep 16 21:52:53.702865 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:52:53.702825    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4ed3a0_8540_4118_95b3_0b8cd252a0ba.slice/crio-9d7aab6cde6694e8198c95fe3ec87343bdf4ee73723538e92d627f3c53c64378 WatchSource:0}: Error finding container 9d7aab6cde6694e8198c95fe3ec87343bdf4ee73723538e92d627f3c53c64378: Status 404 returned error can't find the container with id 9d7aab6cde6694e8198c95fe3ec87343bdf4ee73723538e92d627f3c53c64378
Sep 16 21:52:54.084982 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:54.084946    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh" event={"ID":"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba","Type":"ContainerStarted","Data":"1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5"}
Sep 16 21:52:54.085436 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:54.084999    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh" event={"ID":"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba","Type":"ContainerStarted","Data":"9d7aab6cde6694e8198c95fe3ec87343bdf4ee73723538e92d627f3c53c64378"}
Sep 16 21:52:54.085436 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:54.085062    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:52:54.086778 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:54.086756    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2" event={"ID":"c2430b5e-0754-405e-afdf-571515bfcee5","Type":"ContainerStarted","Data":"a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3"}
Sep 16 21:52:54.086864 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:54.086805    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:52:54.086864 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:54.086826    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2" event={"ID":"c2430b5e-0754-405e-afdf-571515bfcee5","Type":"ContainerStarted","Data":"4b519fda43a4d6d935ef3167fdc78f6e01dfd0243669a14837f62a51e588a8ae"}
Sep 16 21:52:54.158368 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:54.158310    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh" podStartSLOduration=1.158293346 podStartE2EDuration="1.158293346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:52:54.11153429 +0000 UTC m=+1813.497217393" watchObservedRunningTime="2026-09-16 21:52:54.158293346 +0000 UTC m=+1813.543976433"
Sep 16 21:52:59.294780 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:59.294722    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2" podStartSLOduration=6.294705769 podStartE2EDuration="6.294705769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:52:54.155880634 +0000 UTC m=+1813.541563724" watchObservedRunningTime="2026-09-16 21:52:59.294705769 +0000 UTC m=+1818.680388928"
Sep 16 21:52:59.296635 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:52:59.296613    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:53:05.094269 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:05.094232    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:53:05.094759 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:05.094589    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:53:30.909519 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:30.909486    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"]
Sep 16 21:53:30.912113 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:30.909741    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2" podUID="c2430b5e-0754-405e-afdf-571515bfcee5" containerName="payload-processing" containerID="cri-o://a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3" gracePeriod=30
Sep 16 21:53:30.916928 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:30.916901    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"]
Sep 16 21:53:30.917249 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:30.917137    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh" podUID="7a4ed3a0-8540-4118-95b3-0b8cd252a0ba" containerName="payload-pre-processing" containerID="cri-o://1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5" gracePeriod=30
Sep 16 21:53:31.170943 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.170888    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:53:31.197715 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.197691    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:53:31.299600 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.299567    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8kck\" (UniqueName: \"kubernetes.io/projected/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-kube-api-access-v8kck\") pod \"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba\" (UID: \"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba\") "
Sep 16 21:53:31.299784 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.299641    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2430b5e-0754-405e-afdf-571515bfcee5-plugins-config-volume\") pod \"c2430b5e-0754-405e-afdf-571515bfcee5\" (UID: \"c2430b5e-0754-405e-afdf-571515bfcee5\") "
Sep 16 21:53:31.299784 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.299667    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjfhm\" (UniqueName: \"kubernetes.io/projected/c2430b5e-0754-405e-afdf-571515bfcee5-kube-api-access-cjfhm\") pod \"c2430b5e-0754-405e-afdf-571515bfcee5\" (UID: \"c2430b5e-0754-405e-afdf-571515bfcee5\") "
Sep 16 21:53:31.299784 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.299700    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-plugins-config-volume\") pod \"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba\" (UID: \"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba\") "
Sep 16 21:53:31.300077 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.300035    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2430b5e-0754-405e-afdf-571515bfcee5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2430b5e-0754-405e-afdf-571515bfcee5" (UID: "c2430b5e-0754-405e-afdf-571515bfcee5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:53:31.300202 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.300085    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a4ed3a0-8540-4118-95b3-0b8cd252a0ba" (UID: "7a4ed3a0-8540-4118-95b3-0b8cd252a0ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:53:31.301678 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.301654    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-kube-api-access-v8kck" (OuterVolumeSpecName: "kube-api-access-v8kck") pod "7a4ed3a0-8540-4118-95b3-0b8cd252a0ba" (UID: "7a4ed3a0-8540-4118-95b3-0b8cd252a0ba"). InnerVolumeSpecName "kube-api-access-v8kck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:53:31.301764 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.301733    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2430b5e-0754-405e-afdf-571515bfcee5-kube-api-access-cjfhm" (OuterVolumeSpecName: "kube-api-access-cjfhm") pod "c2430b5e-0754-405e-afdf-571515bfcee5" (UID: "c2430b5e-0754-405e-afdf-571515bfcee5"). InnerVolumeSpecName "kube-api-access-cjfhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:53:31.312480 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.312455    2567 generic.go:356] "Generic (PLEG): container finished" podID="7a4ed3a0-8540-4118-95b3-0b8cd252a0ba" containerID="1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5" exitCode=0
Sep 16 21:53:31.312583 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.312495    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"
Sep 16 21:53:31.312583 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.312535    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh" event={"ID":"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba","Type":"ContainerDied","Data":"1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5"}
Sep 16 21:53:31.312697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.312594    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh" event={"ID":"7a4ed3a0-8540-4118-95b3-0b8cd252a0ba","Type":"ContainerDied","Data":"9d7aab6cde6694e8198c95fe3ec87343bdf4ee73723538e92d627f3c53c64378"}
Sep 16 21:53:31.312697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.312627    2567 scope.go:117] "RemoveContainer" containerID="1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5"
Sep 16 21:53:31.313962 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.313943    2567 generic.go:356] "Generic (PLEG): container finished" podID="c2430b5e-0754-405e-afdf-571515bfcee5" containerID="a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3" exitCode=0
Sep 16 21:53:31.314064 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.313989    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"
Sep 16 21:53:31.314064 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.314002    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2" event={"ID":"c2430b5e-0754-405e-afdf-571515bfcee5","Type":"ContainerDied","Data":"a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3"}
Sep 16 21:53:31.314064 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.314033    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2" event={"ID":"c2430b5e-0754-405e-afdf-571515bfcee5","Type":"ContainerDied","Data":"4b519fda43a4d6d935ef3167fdc78f6e01dfd0243669a14837f62a51e588a8ae"}
Sep 16 21:53:31.323682 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.323659    2567 scope.go:117] "RemoveContainer" containerID="1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5"
Sep 16 21:53:31.324159 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:53:31.324135    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5\": container with ID starting with 1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5 not found: ID does not exist" containerID="1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5"
Sep 16 21:53:31.324258 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.324165    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5"} err="failed to get container status \"1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5\": rpc error: code = NotFound desc = could not find container \"1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5\": container with ID starting with 1fbe938dc071de29d5a6d10299e921b1dd7532b6a80d9faf699b88122c1222c5 not found: ID does not exist"
Sep 16 21:53:31.324258 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.324188    2567 scope.go:117] "RemoveContainer" containerID="a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3"
Sep 16 21:53:31.334390 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.334370    2567 scope.go:117] "RemoveContainer" containerID="a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3"
Sep 16 21:53:31.334688 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:53:31.334669    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3\": container with ID starting with a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3 not found: ID does not exist" containerID="a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3"
Sep 16 21:53:31.334752 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.334698    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3"} err="failed to get container status \"a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3\": rpc error: code = NotFound desc = could not find container \"a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3\": container with ID starting with a40f624cece15a52315f9d637f945680f88e37c856b4e1f26ccb104c17cf83e3 not found: ID does not exist"
Sep 16 21:53:31.335622 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.335602    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"]
Sep 16 21:53:31.339053 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.339030    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dcbfbc-5d949b9d66-8c9hh"]
Sep 16 21:53:31.348848 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.348829    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"]
Sep 16 21:53:31.353011 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.352992    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dcbfbc-5b785c676c-hkms2"]
Sep 16 21:53:31.401092 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.401068    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjfhm\" (UniqueName: \"kubernetes.io/projected/c2430b5e-0754-405e-afdf-571515bfcee5-kube-api-access-cjfhm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:53:31.401092 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.401090    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:53:31.401212 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.401099    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8kck\" (UniqueName: \"kubernetes.io/projected/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba-kube-api-access-v8kck\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:53:31.401212 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:31.401108    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2430b5e-0754-405e-afdf-571515bfcee5-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:53:33.109032 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:33.109000    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a4ed3a0-8540-4118-95b3-0b8cd252a0ba" path="/var/lib/kubelet/pods/7a4ed3a0-8540-4118-95b3-0b8cd252a0ba/volumes"
Sep 16 21:53:33.109413 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:33.109379    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2430b5e-0754-405e-afdf-571515bfcee5" path="/var/lib/kubelet/pods/c2430b5e-0754-405e-afdf-571515bfcee5/volumes"
Sep 16 21:53:35.680819 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:35.680782    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:53:54.283304 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:54.283267    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:53:56.799658 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:56.799622    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"]
Sep 16 21:53:56.800137 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:56.799882    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2" podUID="e626ed97-e7b3-402c-a6cf-ad17d0bbd344" containerName="payload-processing" containerID="cri-o://c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094" gracePeriod=30
Sep 16 21:53:56.808490 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:56.808164    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"]
Sep 16 21:53:56.808621 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:56.808519    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt" podUID="b6192bb3-09bf-4341-8520-55e0b2fb0170" containerName="payload-pre-processing" containerID="cri-o://743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f" gracePeriod=30
Sep 16 21:53:57.094141 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.094119    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:53:57.097166 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.097145    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:53:57.235218 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.235186    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-plugins-config-volume\") pod \"e626ed97-e7b3-402c-a6cf-ad17d0bbd344\" (UID: \"e626ed97-e7b3-402c-a6cf-ad17d0bbd344\") "
Sep 16 21:53:57.235439 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.235250    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkfmf\" (UniqueName: \"kubernetes.io/projected/b6192bb3-09bf-4341-8520-55e0b2fb0170-kube-api-access-tkfmf\") pod \"b6192bb3-09bf-4341-8520-55e0b2fb0170\" (UID: \"b6192bb3-09bf-4341-8520-55e0b2fb0170\") "
Sep 16 21:53:57.235439 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.235287    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2t27\" (UniqueName: \"kubernetes.io/projected/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-kube-api-access-k2t27\") pod \"e626ed97-e7b3-402c-a6cf-ad17d0bbd344\" (UID: \"e626ed97-e7b3-402c-a6cf-ad17d0bbd344\") "
Sep 16 21:53:57.235439 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.235348    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6192bb3-09bf-4341-8520-55e0b2fb0170-plugins-config-volume\") pod \"b6192bb3-09bf-4341-8520-55e0b2fb0170\" (UID: \"b6192bb3-09bf-4341-8520-55e0b2fb0170\") "
Sep 16 21:53:57.235657 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.235628    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e626ed97-e7b3-402c-a6cf-ad17d0bbd344" (UID: "e626ed97-e7b3-402c-a6cf-ad17d0bbd344"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:53:57.235741 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.235724    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:53:57.236031 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.236004    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6192bb3-09bf-4341-8520-55e0b2fb0170-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6192bb3-09bf-4341-8520-55e0b2fb0170" (UID: "b6192bb3-09bf-4341-8520-55e0b2fb0170"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:53:57.237522 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.237497    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6192bb3-09bf-4341-8520-55e0b2fb0170-kube-api-access-tkfmf" (OuterVolumeSpecName: "kube-api-access-tkfmf") pod "b6192bb3-09bf-4341-8520-55e0b2fb0170" (UID: "b6192bb3-09bf-4341-8520-55e0b2fb0170"). InnerVolumeSpecName "kube-api-access-tkfmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:53:57.237619 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.237602    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-kube-api-access-k2t27" (OuterVolumeSpecName: "kube-api-access-k2t27") pod "e626ed97-e7b3-402c-a6cf-ad17d0bbd344" (UID: "e626ed97-e7b3-402c-a6cf-ad17d0bbd344"). InnerVolumeSpecName "kube-api-access-k2t27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:53:57.336197 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.336124    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkfmf\" (UniqueName: \"kubernetes.io/projected/b6192bb3-09bf-4341-8520-55e0b2fb0170-kube-api-access-tkfmf\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:53:57.336197 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.336157    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2t27\" (UniqueName: \"kubernetes.io/projected/e626ed97-e7b3-402c-a6cf-ad17d0bbd344-kube-api-access-k2t27\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:53:57.336197 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.336170    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6192bb3-09bf-4341-8520-55e0b2fb0170-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:53:57.475461 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.475422    2567 generic.go:356] "Generic (PLEG): container finished" podID="b6192bb3-09bf-4341-8520-55e0b2fb0170" containerID="743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f" exitCode=0
Sep 16 21:53:57.475637 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.475464    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"
Sep 16 21:53:57.475637 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.475486    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt" event={"ID":"b6192bb3-09bf-4341-8520-55e0b2fb0170","Type":"ContainerDied","Data":"743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f"}
Sep 16 21:53:57.475637 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.475539    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt" event={"ID":"b6192bb3-09bf-4341-8520-55e0b2fb0170","Type":"ContainerDied","Data":"06e5ec048730295521909f626d4b44adc7dc7cc7079e7489911e6f8b9a5f5111"}
Sep 16 21:53:57.475637 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.475557    2567 scope.go:117] "RemoveContainer" containerID="743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f"
Sep 16 21:53:57.476894 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.476870    2567 generic.go:356] "Generic (PLEG): container finished" podID="e626ed97-e7b3-402c-a6cf-ad17d0bbd344" containerID="c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094" exitCode=0
Sep 16 21:53:57.477013 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.476917    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"
Sep 16 21:53:57.477013 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.476942    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2" event={"ID":"e626ed97-e7b3-402c-a6cf-ad17d0bbd344","Type":"ContainerDied","Data":"c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094"}
Sep 16 21:53:57.477013 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.476974    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2" event={"ID":"e626ed97-e7b3-402c-a6cf-ad17d0bbd344","Type":"ContainerDied","Data":"5abf992e03547d766ac3a65a68c1e0635aa89cab4c1e771613331f05ef53be44"}
Sep 16 21:53:57.487961 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.487940    2567 scope.go:117] "RemoveContainer" containerID="743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f"
Sep 16 21:53:57.488299 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:53:57.488275    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f\": container with ID starting with 743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f not found: ID does not exist" containerID="743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f"
Sep 16 21:53:57.488477 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.488453    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f"} err="failed to get container status \"743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f\": rpc error: code = NotFound desc = could not find container \"743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f\": container with ID starting with 743cf656f6faa2c03e7f90374c4c1981849297b711b9e811d33040c6fe1ed15f not found: ID does not exist"
Sep 16 21:53:57.488610 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.488591    2567 scope.go:117] "RemoveContainer" containerID="c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094"
Sep 16 21:53:57.499017 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.498993    2567 scope.go:117] "RemoveContainer" containerID="c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094"
Sep 16 21:53:57.499301 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:53:57.499281    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094\": container with ID starting with c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094 not found: ID does not exist" containerID="c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094"
Sep 16 21:53:57.499365 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.499308    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094"} err="failed to get container status \"c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094\": rpc error: code = NotFound desc = could not find container \"c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094\": container with ID starting with c30d16ec00a89ab75bfa795ff873dcdfb082c9287b912e98bdfc00c488815094 not found: ID does not exist"
Sep 16 21:53:57.506116 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.506095    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"]
Sep 16 21:53:57.510314 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.510295    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad0222-74ccc8db79-7ccx2"]
Sep 16 21:53:57.519310 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.519274    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"]
Sep 16 21:53:57.524066 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:57.524046    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad0222-c59474794-rh2bt"]
Sep 16 21:53:59.109488 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:59.109447    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6192bb3-09bf-4341-8520-55e0b2fb0170" path="/var/lib/kubelet/pods/b6192bb3-09bf-4341-8520-55e0b2fb0170/volumes"
Sep 16 21:53:59.110008 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:53:59.109984    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e626ed97-e7b3-402c-a6cf-ad17d0bbd344" path="/var/lib/kubelet/pods/e626ed97-e7b3-402c-a6cf-ad17d0bbd344/volumes"
Sep 16 21:54:07.503192 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:07.503111    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:54:46.188293 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188256    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"]
Sep 16 21:54:46.188749 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188734    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e626ed97-e7b3-402c-a6cf-ad17d0bbd344" containerName="payload-processing"
Sep 16 21:54:46.188795 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188751    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e626ed97-e7b3-402c-a6cf-ad17d0bbd344" containerName="payload-processing"
Sep 16 21:54:46.188795 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188759    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6192bb3-09bf-4341-8520-55e0b2fb0170" containerName="payload-pre-processing"
Sep 16 21:54:46.188795 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188766    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6192bb3-09bf-4341-8520-55e0b2fb0170" containerName="payload-pre-processing"
Sep 16 21:54:46.188795 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188789    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2430b5e-0754-405e-afdf-571515bfcee5" containerName="payload-processing"
Sep 16 21:54:46.188795 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188795    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2430b5e-0754-405e-afdf-571515bfcee5" containerName="payload-processing"
Sep 16 21:54:46.188952 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188802    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a4ed3a0-8540-4118-95b3-0b8cd252a0ba" containerName="payload-pre-processing"
Sep 16 21:54:46.188952 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188808    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a4ed3a0-8540-4118-95b3-0b8cd252a0ba" containerName="payload-pre-processing"
Sep 16 21:54:46.188952 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188870    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6192bb3-09bf-4341-8520-55e0b2fb0170" containerName="payload-pre-processing"
Sep 16 21:54:46.188952 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188878    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2430b5e-0754-405e-afdf-571515bfcee5" containerName="payload-processing"
Sep 16 21:54:46.188952 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188886    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e626ed97-e7b3-402c-a6cf-ad17d0bbd344" containerName="payload-processing"
Sep 16 21:54:46.188952 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.188894    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a4ed3a0-8540-4118-95b3-0b8cd252a0ba" containerName="payload-pre-processing"
Sep 16 21:54:46.192035 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.192019    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"
Sep 16 21:54:46.194253 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.194235    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:54:46.194590 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.194574    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:54:46.203432 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.203386    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"]
Sep 16 21:54:46.228257 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.228221    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df639b9-9c7b-4833-a5ac-4b97a66ccbe7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl\" (UID: \"0df639b9-9c7b-4833-a5ac-4b97a66ccbe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"
Sep 16 21:54:46.329144 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.329114    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df639b9-9c7b-4833-a5ac-4b97a66ccbe7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl\" (UID: \"0df639b9-9c7b-4833-a5ac-4b97a66ccbe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"
Sep 16 21:54:46.329802 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.329768    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df639b9-9c7b-4833-a5ac-4b97a66ccbe7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl\" (UID: \"0df639b9-9c7b-4833-a5ac-4b97a66ccbe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"
Sep 16 21:54:46.502302 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.502270    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"
Sep 16 21:54:46.781070 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:46.781044    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"]
Sep 16 21:54:46.782813 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:54:46.782781    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df639b9_9c7b_4833_a5ac_4b97a66ccbe7.slice/crio-fc50b923e74da833a97e8526f5bb95c515a251d117af582ed5f0b14887658635 WatchSource:0}: Error finding container fc50b923e74da833a97e8526f5bb95c515a251d117af582ed5f0b14887658635: Status 404 returned error can't find the container with id fc50b923e74da833a97e8526f5bb95c515a251d117af582ed5f0b14887658635
Sep 16 21:54:47.772450 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:47.772391    2567 generic.go:356] "Generic (PLEG): container finished" podID="0df639b9-9c7b-4833-a5ac-4b97a66ccbe7" containerID="0ab6b9b191dc5c94e07dc8f9f586c7d2e18afaf11246479509d3d6f2dffd94af" exitCode=0
Sep 16 21:54:47.772849 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:47.772497    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl" event={"ID":"0df639b9-9c7b-4833-a5ac-4b97a66ccbe7","Type":"ContainerDied","Data":"0ab6b9b191dc5c94e07dc8f9f586c7d2e18afaf11246479509d3d6f2dffd94af"}
Sep 16 21:54:47.772849 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:47.772573    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl" event={"ID":"0df639b9-9c7b-4833-a5ac-4b97a66ccbe7","Type":"ContainerStarted","Data":"fc50b923e74da833a97e8526f5bb95c515a251d117af582ed5f0b14887658635"}
Sep 16 21:54:48.913134 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:48.913109    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"
Sep 16 21:54:48.951936 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:48.951910    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df639b9-9c7b-4833-a5ac-4b97a66ccbe7-maas-api-service-ca\") pod \"0df639b9-9c7b-4833-a5ac-4b97a66ccbe7\" (UID: \"0df639b9-9c7b-4833-a5ac-4b97a66ccbe7\") "
Sep 16 21:54:48.952237 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:48.952214    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df639b9-9c7b-4833-a5ac-4b97a66ccbe7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0df639b9-9c7b-4833-a5ac-4b97a66ccbe7" (UID: "0df639b9-9c7b-4833-a5ac-4b97a66ccbe7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:54:49.052562 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:49.052501    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df639b9-9c7b-4833-a5ac-4b97a66ccbe7-maas-api-service-ca\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:54:49.785517 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:49.785485    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"
Sep 16 21:54:49.785517 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:49.785495    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl" event={"ID":"0df639b9-9c7b-4833-a5ac-4b97a66ccbe7","Type":"ContainerDied","Data":"fc50b923e74da833a97e8526f5bb95c515a251d117af582ed5f0b14887658635"}
Sep 16 21:54:49.785721 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:49.785543    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc50b923e74da833a97e8526f5bb95c515a251d117af582ed5f0b14887658635"
Sep 16 21:54:56.789781 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:56.789744    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"]
Sep 16 21:54:56.790216 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:56.789948    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9" podUID="8d14c8e0-41b7-4bfd-905b-8988dd0555c3" containerName="payload-processing" containerID="cri-o://85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c" gracePeriod=30
Sep 16 21:54:56.798972 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:56.798946    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"]
Sep 16 21:54:56.799157 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:56.799132    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49" podUID="21e01938-b669-467e-9f3e-5489019b56fd" containerName="payload-pre-processing" containerID="cri-o://a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2" gracePeriod=30
Sep 16 21:54:56.951055 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:56.951016    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"]
Sep 16 21:54:56.953371 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:56.953341    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03eca6-hlpdl"]
Sep 16 21:54:57.073593 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.073565    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:54:57.076372 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.076354    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:54:57.111580 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.111543    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0df639b9-9c7b-4833-a5ac-4b97a66ccbe7" path="/var/lib/kubelet/pods/0df639b9-9c7b-4833-a5ac-4b97a66ccbe7/volumes"
Sep 16 21:54:57.118114 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.118085    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-plugins-config-volume\") pod \"8d14c8e0-41b7-4bfd-905b-8988dd0555c3\" (UID: \"8d14c8e0-41b7-4bfd-905b-8988dd0555c3\") "
Sep 16 21:54:57.118254 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.118183    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg2st\" (UniqueName: \"kubernetes.io/projected/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-kube-api-access-pg2st\") pod \"8d14c8e0-41b7-4bfd-905b-8988dd0555c3\" (UID: \"8d14c8e0-41b7-4bfd-905b-8988dd0555c3\") "
Sep 16 21:54:57.118254 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.118203    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e01938-b669-467e-9f3e-5489019b56fd-plugins-config-volume\") pod \"21e01938-b669-467e-9f3e-5489019b56fd\" (UID: \"21e01938-b669-467e-9f3e-5489019b56fd\") "
Sep 16 21:54:57.118254 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.118236    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx6n7\" (UniqueName: \"kubernetes.io/projected/21e01938-b669-467e-9f3e-5489019b56fd-kube-api-access-bx6n7\") pod \"21e01938-b669-467e-9f3e-5489019b56fd\" (UID: \"21e01938-b669-467e-9f3e-5489019b56fd\") "
Sep 16 21:54:57.118499 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.118475    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d14c8e0-41b7-4bfd-905b-8988dd0555c3" (UID: "8d14c8e0-41b7-4bfd-905b-8988dd0555c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:54:57.118600 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.118571    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:54:57.118704 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.118680    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e01938-b669-467e-9f3e-5489019b56fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21e01938-b669-467e-9f3e-5489019b56fd" (UID: "21e01938-b669-467e-9f3e-5489019b56fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:54:57.120445 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.120414    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-kube-api-access-pg2st" (OuterVolumeSpecName: "kube-api-access-pg2st") pod "8d14c8e0-41b7-4bfd-905b-8988dd0555c3" (UID: "8d14c8e0-41b7-4bfd-905b-8988dd0555c3"). InnerVolumeSpecName "kube-api-access-pg2st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:54:57.120532 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.120444    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e01938-b669-467e-9f3e-5489019b56fd-kube-api-access-bx6n7" (OuterVolumeSpecName: "kube-api-access-bx6n7") pod "21e01938-b669-467e-9f3e-5489019b56fd" (UID: "21e01938-b669-467e-9f3e-5489019b56fd"). InnerVolumeSpecName "kube-api-access-bx6n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:54:57.220014 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.219976    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg2st\" (UniqueName: \"kubernetes.io/projected/8d14c8e0-41b7-4bfd-905b-8988dd0555c3-kube-api-access-pg2st\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:54:57.220014 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.220006    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e01938-b669-467e-9f3e-5489019b56fd-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:54:57.220014 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.220022    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx6n7\" (UniqueName: \"kubernetes.io/projected/21e01938-b669-467e-9f3e-5489019b56fd-kube-api-access-bx6n7\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:54:57.832859 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.832820    2567 generic.go:356] "Generic (PLEG): container finished" podID="8d14c8e0-41b7-4bfd-905b-8988dd0555c3" containerID="85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c" exitCode=0
Sep 16 21:54:57.833300 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.832866    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"
Sep 16 21:54:57.833300 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.832909    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9" event={"ID":"8d14c8e0-41b7-4bfd-905b-8988dd0555c3","Type":"ContainerDied","Data":"85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c"}
Sep 16 21:54:57.833300 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.832963    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9" event={"ID":"8d14c8e0-41b7-4bfd-905b-8988dd0555c3","Type":"ContainerDied","Data":"7580c40a3b46f7b8b324ee10497297febf3fa043f5a59908c61318e636506648"}
Sep 16 21:54:57.833300 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.832989    2567 scope.go:117] "RemoveContainer" containerID="85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c"
Sep 16 21:54:57.834679 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.834645    2567 generic.go:356] "Generic (PLEG): container finished" podID="21e01938-b669-467e-9f3e-5489019b56fd" containerID="a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2" exitCode=0
Sep 16 21:54:57.834769 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.834689    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49" event={"ID":"21e01938-b669-467e-9f3e-5489019b56fd","Type":"ContainerDied","Data":"a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2"}
Sep 16 21:54:57.834769 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.834696    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"
Sep 16 21:54:57.834769 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.834730    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49" event={"ID":"21e01938-b669-467e-9f3e-5489019b56fd","Type":"ContainerDied","Data":"991334a4569f841149abe26924d9615f92aa72b5e9642843498c3dd1d10cbfb0"}
Sep 16 21:54:57.844161 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.844142    2567 scope.go:117] "RemoveContainer" containerID="85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c"
Sep 16 21:54:57.844486 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:54:57.844459    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c\": container with ID starting with 85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c not found: ID does not exist" containerID="85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c"
Sep 16 21:54:57.844572 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.844499    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c"} err="failed to get container status \"85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c\": rpc error: code = NotFound desc = could not find container \"85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c\": container with ID starting with 85a95ca4c1e9dd1d7835f2737b84d34b8932dde99f43ba13e2055f82370a849c not found: ID does not exist"
Sep 16 21:54:57.844572 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.844525    2567 scope.go:117] "RemoveContainer" containerID="a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2"
Sep 16 21:54:57.855259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.855242    2567 scope.go:117] "RemoveContainer" containerID="a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2"
Sep 16 21:54:57.855544 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:54:57.855524    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2\": container with ID starting with a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2 not found: ID does not exist" containerID="a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2"
Sep 16 21:54:57.855616 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.855556    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2"} err="failed to get container status \"a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2\": rpc error: code = NotFound desc = could not find container \"a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2\": container with ID starting with a5ccd92e567b30711a523a352332a6efd8488345caa8f6d1690cf72cad1aa7a2 not found: ID does not exist"
Sep 16 21:54:57.868969 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.868942    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"]
Sep 16 21:54:57.876758 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.876724    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03eca6-7b479bbb54-4kj49"]
Sep 16 21:54:57.885481 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.885461    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"]
Sep 16 21:54:57.890898 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:57.890878    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03eca6-6558849f7f-2lrv9"]
Sep 16 21:54:59.109018 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:59.108977    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21e01938-b669-467e-9f3e-5489019b56fd" path="/var/lib/kubelet/pods/21e01938-b669-467e-9f3e-5489019b56fd/volumes"
Sep 16 21:54:59.109438 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:54:59.109422    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d14c8e0-41b7-4bfd-905b-8988dd0555c3" path="/var/lib/kubelet/pods/8d14c8e0-41b7-4bfd-905b-8988dd0555c3/volumes"
Sep 16 21:55:00.084089 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:00.084048    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:55:03.914337 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.914300    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"]
Sep 16 21:55:03.914925 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.914903    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21e01938-b669-467e-9f3e-5489019b56fd" containerName="payload-pre-processing"
Sep 16 21:55:03.915010 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.914928    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="21e01938-b669-467e-9f3e-5489019b56fd" containerName="payload-pre-processing"
Sep 16 21:55:03.915010 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.914946    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d14c8e0-41b7-4bfd-905b-8988dd0555c3" containerName="payload-processing"
Sep 16 21:55:03.915010 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.914955    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d14c8e0-41b7-4bfd-905b-8988dd0555c3" containerName="payload-processing"
Sep 16 21:55:03.915010 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.914969    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df639b9-9c7b-4833-a5ac-4b97a66ccbe7" containerName="revoke-keys"
Sep 16 21:55:03.915010 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.914978    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df639b9-9c7b-4833-a5ac-4b97a66ccbe7" containerName="revoke-keys"
Sep 16 21:55:03.915259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.915076    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="21e01938-b669-467e-9f3e-5489019b56fd" containerName="payload-pre-processing"
Sep 16 21:55:03.915259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.915089    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df639b9-9c7b-4833-a5ac-4b97a66ccbe7" containerName="revoke-keys"
Sep 16 21:55:03.915259 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.915103    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d14c8e0-41b7-4bfd-905b-8988dd0555c3" containerName="payload-processing"
Sep 16 21:55:03.919697 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.919675    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"
Sep 16 21:55:03.921935 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.921909    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:55:03.922231 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.922211    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:55:03.924439 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.924417    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"]
Sep 16 21:55:03.980202 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:03.980174    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dce157d-91a4-4d82-9505-7cee2a16f2ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh\" (UID: \"1dce157d-91a4-4d82-9505-7cee2a16f2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"
Sep 16 21:55:04.081197 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:04.081170    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dce157d-91a4-4d82-9505-7cee2a16f2ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh\" (UID: \"1dce157d-91a4-4d82-9505-7cee2a16f2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"
Sep 16 21:55:04.081814 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:04.081795    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dce157d-91a4-4d82-9505-7cee2a16f2ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh\" (UID: \"1dce157d-91a4-4d82-9505-7cee2a16f2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"
Sep 16 21:55:04.231860 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:04.231834    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"
Sep 16 21:55:04.504950 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:04.504917    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"]
Sep 16 21:55:04.506502 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:55:04.506472    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dce157d_91a4_4d82_9505_7cee2a16f2ce.slice/crio-04a1c43b7458809ad6d1defb664040981e38d5aa3c87ec6615ddaa991fb3815b WatchSource:0}: Error finding container 04a1c43b7458809ad6d1defb664040981e38d5aa3c87ec6615ddaa991fb3815b: Status 404 returned error can't find the container with id 04a1c43b7458809ad6d1defb664040981e38d5aa3c87ec6615ddaa991fb3815b
Sep 16 21:55:04.876727 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:04.876695    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh" event={"ID":"1dce157d-91a4-4d82-9505-7cee2a16f2ce","Type":"ContainerStarted","Data":"9d489b30d5c977b69eae1b579fdee321f0d3cca66b54b17d378dc1854fb1cfe4"}
Sep 16 21:55:04.876874 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:04.876746    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh" event={"ID":"1dce157d-91a4-4d82-9505-7cee2a16f2ce","Type":"ContainerStarted","Data":"04a1c43b7458809ad6d1defb664040981e38d5aa3c87ec6615ddaa991fb3815b"}
Sep 16 21:55:04.891305 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:04.891259    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh" podStartSLOduration=1.891245934 podStartE2EDuration="1.891245934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:55:04.889441439 +0000 UTC m=+1944.275124544" watchObservedRunningTime="2026-09-16 21:55:04.891245934 +0000 UTC m=+1944.276929018"
Sep 16 21:55:05.884029 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:05.883994    2567 generic.go:356] "Generic (PLEG): container finished" podID="1dce157d-91a4-4d82-9505-7cee2a16f2ce" containerID="9d489b30d5c977b69eae1b579fdee321f0d3cca66b54b17d378dc1854fb1cfe4" exitCode=0
Sep 16 21:55:05.884431 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:05.884085    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh" event={"ID":"1dce157d-91a4-4d82-9505-7cee2a16f2ce","Type":"ContainerDied","Data":"9d489b30d5c977b69eae1b579fdee321f0d3cca66b54b17d378dc1854fb1cfe4"}
Sep 16 21:55:07.022764 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:07.022739    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"
Sep 16 21:55:07.108186 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:07.108158    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dce157d-91a4-4d82-9505-7cee2a16f2ce-maas-api-service-ca\") pod \"1dce157d-91a4-4d82-9505-7cee2a16f2ce\" (UID: \"1dce157d-91a4-4d82-9505-7cee2a16f2ce\") "
Sep 16 21:55:07.108446 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:07.108422    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dce157d-91a4-4d82-9505-7cee2a16f2ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1dce157d-91a4-4d82-9505-7cee2a16f2ce" (UID: "1dce157d-91a4-4d82-9505-7cee2a16f2ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:55:07.208854 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:07.208829    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dce157d-91a4-4d82-9505-7cee2a16f2ce-maas-api-service-ca\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:55:07.897247 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:07.897206    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh" event={"ID":"1dce157d-91a4-4d82-9505-7cee2a16f2ce","Type":"ContainerDied","Data":"04a1c43b7458809ad6d1defb664040981e38d5aa3c87ec6615ddaa991fb3815b"}
Sep 16 21:55:07.897519 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:07.897291    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04a1c43b7458809ad6d1defb664040981e38d5aa3c87ec6615ddaa991fb3815b"
Sep 16 21:55:07.897519 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:07.897214    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"
Sep 16 21:55:14.404717 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.404673    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"]
Sep 16 21:55:14.405520 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.405481    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k" podUID="e6f00088-7730-4288-b556-fe91c9436ee9" containerName="payload-processing" containerID="cri-o://111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59" gracePeriod=30
Sep 16 21:55:14.414924 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.414889    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"]
Sep 16 21:55:14.415123 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.415100    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s" podUID="cace92e0-5248-4fbe-84b8-534efbf590d8" containerName="payload-pre-processing" containerID="cri-o://5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293" gracePeriod=30
Sep 16 21:55:14.559916 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.559877    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"]
Sep 16 21:55:14.563238 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.563203    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7df81-vq4gh"]
Sep 16 21:55:14.698547 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.698525    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:55:14.701165 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.701144    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:55:14.773549 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.773517    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f00088-7730-4288-b556-fe91c9436ee9-plugins-config-volume\") pod \"e6f00088-7730-4288-b556-fe91c9436ee9\" (UID: \"e6f00088-7730-4288-b556-fe91c9436ee9\") "
Sep 16 21:55:14.773688 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.773565    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8zdp\" (UniqueName: \"kubernetes.io/projected/e6f00088-7730-4288-b556-fe91c9436ee9-kube-api-access-z8zdp\") pod \"e6f00088-7730-4288-b556-fe91c9436ee9\" (UID: \"e6f00088-7730-4288-b556-fe91c9436ee9\") "
Sep 16 21:55:14.773688 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.773589    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cace92e0-5248-4fbe-84b8-534efbf590d8-plugins-config-volume\") pod \"cace92e0-5248-4fbe-84b8-534efbf590d8\" (UID: \"cace92e0-5248-4fbe-84b8-534efbf590d8\") "
Sep 16 21:55:14.773688 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.773644    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjt8d\" (UniqueName: \"kubernetes.io/projected/cace92e0-5248-4fbe-84b8-534efbf590d8-kube-api-access-cjt8d\") pod \"cace92e0-5248-4fbe-84b8-534efbf590d8\" (UID: \"cace92e0-5248-4fbe-84b8-534efbf590d8\") "
Sep 16 21:55:14.773984 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.773956    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f00088-7730-4288-b556-fe91c9436ee9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6f00088-7730-4288-b556-fe91c9436ee9" (UID: "e6f00088-7730-4288-b556-fe91c9436ee9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:55:14.774031 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.773984    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cace92e0-5248-4fbe-84b8-534efbf590d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cace92e0-5248-4fbe-84b8-534efbf590d8" (UID: "cace92e0-5248-4fbe-84b8-534efbf590d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:55:14.775776 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.775753    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f00088-7730-4288-b556-fe91c9436ee9-kube-api-access-z8zdp" (OuterVolumeSpecName: "kube-api-access-z8zdp") pod "e6f00088-7730-4288-b556-fe91c9436ee9" (UID: "e6f00088-7730-4288-b556-fe91c9436ee9"). InnerVolumeSpecName "kube-api-access-z8zdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:14.775875 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.775819    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cace92e0-5248-4fbe-84b8-534efbf590d8-kube-api-access-cjt8d" (OuterVolumeSpecName: "kube-api-access-cjt8d") pod "cace92e0-5248-4fbe-84b8-534efbf590d8" (UID: "cace92e0-5248-4fbe-84b8-534efbf590d8"). InnerVolumeSpecName "kube-api-access-cjt8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:14.874926 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.874896    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjt8d\" (UniqueName: \"kubernetes.io/projected/cace92e0-5248-4fbe-84b8-534efbf590d8-kube-api-access-cjt8d\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:55:14.874926 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.874921    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f00088-7730-4288-b556-fe91c9436ee9-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:55:14.874926 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.874931    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8zdp\" (UniqueName: \"kubernetes.io/projected/e6f00088-7730-4288-b556-fe91c9436ee9-kube-api-access-z8zdp\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:55:14.875103 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.874940    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cace92e0-5248-4fbe-84b8-534efbf590d8-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:55:14.937057 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.936988    2567 generic.go:356] "Generic (PLEG): container finished" podID="cace92e0-5248-4fbe-84b8-534efbf590d8" containerID="5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293" exitCode=0
Sep 16 21:55:14.937057 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.937047    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"
Sep 16 21:55:14.937226 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.937075    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s" event={"ID":"cace92e0-5248-4fbe-84b8-534efbf590d8","Type":"ContainerDied","Data":"5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293"}
Sep 16 21:55:14.937226 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.937170    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s" event={"ID":"cace92e0-5248-4fbe-84b8-534efbf590d8","Type":"ContainerDied","Data":"174ce9bdfc53e16c08bf9853a2761f890a025eb84632ea26dfec07f50f5106f6"}
Sep 16 21:55:14.937226 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.937199    2567 scope.go:117] "RemoveContainer" containerID="5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293"
Sep 16 21:55:14.938581 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.938555    2567 generic.go:356] "Generic (PLEG): container finished" podID="e6f00088-7730-4288-b556-fe91c9436ee9" containerID="111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59" exitCode=0
Sep 16 21:55:14.938709 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.938635    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"
Sep 16 21:55:14.938709 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.938650    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k" event={"ID":"e6f00088-7730-4288-b556-fe91c9436ee9","Type":"ContainerDied","Data":"111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59"}
Sep 16 21:55:14.938709 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.938701    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k" event={"ID":"e6f00088-7730-4288-b556-fe91c9436ee9","Type":"ContainerDied","Data":"48ef44156ca711546f15e8394e521a35e0f9073b2eb1bba941b9be4229d422a1"}
Sep 16 21:55:14.946945 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.946928    2567 scope.go:117] "RemoveContainer" containerID="5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293"
Sep 16 21:55:14.947191 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:55:14.947173    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293\": container with ID starting with 5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293 not found: ID does not exist" containerID="5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293"
Sep 16 21:55:14.947262 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.947198    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293"} err="failed to get container status \"5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293\": rpc error: code = NotFound desc = could not find container \"5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293\": container with ID starting with 5acc0ad507fb72b97f4368f6a4281613cf04aebe1cbe916024ebe2bcc71f4293 not found: ID does not exist"
Sep 16 21:55:14.947262 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.947214    2567 scope.go:117] "RemoveContainer" containerID="111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59"
Sep 16 21:55:14.955116 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.955098    2567 scope.go:117] "RemoveContainer" containerID="111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59"
Sep 16 21:55:14.955359 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:55:14.955338    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59\": container with ID starting with 111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59 not found: ID does not exist" containerID="111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59"
Sep 16 21:55:14.955445 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.955365    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59"} err="failed to get container status \"111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59\": rpc error: code = NotFound desc = could not find container \"111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59\": container with ID starting with 111c93a0671bc0c362f0fc9147fbfe96e647c0abbeb6a2da5eb0ca5ac4e85a59 not found: ID does not exist"
Sep 16 21:55:14.961153 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.961132    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"]
Sep 16 21:55:14.963604 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.963585    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7df81-7d68c6ff77-52h2k"]
Sep 16 21:55:14.971041 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.971022    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"]
Sep 16 21:55:14.976638 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:14.976619    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7df81-78cf4c7b46-c4s4s"]
Sep 16 21:55:15.109892 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:15.109862    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dce157d-91a4-4d82-9505-7cee2a16f2ce" path="/var/lib/kubelet/pods/1dce157d-91a4-4d82-9505-7cee2a16f2ce/volumes"
Sep 16 21:55:15.110327 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:15.110305    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cace92e0-5248-4fbe-84b8-534efbf590d8" path="/var/lib/kubelet/pods/cace92e0-5248-4fbe-84b8-534efbf590d8/volumes"
Sep 16 21:55:15.110729 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:15.110710    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6f00088-7730-4288-b556-fe91c9436ee9" path="/var/lib/kubelet/pods/e6f00088-7730-4288-b556-fe91c9436ee9/volumes"
Sep 16 21:55:17.788650 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:17.788613    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:55:32.029417 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:32.029355    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp"]
Sep 16 21:55:32.031993 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:32.031966    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1b68db-29826585-6bqhp"]
Sep 16 21:55:32.059327 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:32.059298    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"]
Sep 16 21:55:32.059560 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:32.059514    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6" podUID="f2108207-6b4e-4e1a-87b9-de4e665af470" containerName="payload-pre-processing" containerID="cri-o://4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273" gracePeriod=30
Sep 16 21:55:32.318828 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:32.318795    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:55:32.417013 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:32.416983    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgdhj\" (UniqueName: \"kubernetes.io/projected/f2108207-6b4e-4e1a-87b9-de4e665af470-kube-api-access-zgdhj\") pod \"f2108207-6b4e-4e1a-87b9-de4e665af470\" (UID: \"f2108207-6b4e-4e1a-87b9-de4e665af470\") "
Sep 16 21:55:32.417013 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:32.417018    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2108207-6b4e-4e1a-87b9-de4e665af470-plugins-config-volume\") pod \"f2108207-6b4e-4e1a-87b9-de4e665af470\" (UID: \"f2108207-6b4e-4e1a-87b9-de4e665af470\") "
Sep 16 21:55:32.417464 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:32.417437    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2108207-6b4e-4e1a-87b9-de4e665af470-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2108207-6b4e-4e1a-87b9-de4e665af470" (UID: "f2108207-6b4e-4e1a-87b9-de4e665af470"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:55:32.418969 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:32.418947    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2108207-6b4e-4e1a-87b9-de4e665af470-kube-api-access-zgdhj" (OuterVolumeSpecName: "kube-api-access-zgdhj") pod "f2108207-6b4e-4e1a-87b9-de4e665af470" (UID: "f2108207-6b4e-4e1a-87b9-de4e665af470"). InnerVolumeSpecName "kube-api-access-zgdhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:32.517695 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:32.517667    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgdhj\" (UniqueName: \"kubernetes.io/projected/f2108207-6b4e-4e1a-87b9-de4e665af470-kube-api-access-zgdhj\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:55:32.517695 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:32.517697    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2108207-6b4e-4e1a-87b9-de4e665af470-plugins-config-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:55:33.050193 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:33.050156    2567 generic.go:356] "Generic (PLEG): container finished" podID="f2108207-6b4e-4e1a-87b9-de4e665af470" containerID="4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273" exitCode=0
Sep 16 21:55:33.050684 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:33.050196    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"
Sep 16 21:55:33.050684 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:33.050230    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6" event={"ID":"f2108207-6b4e-4e1a-87b9-de4e665af470","Type":"ContainerDied","Data":"4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273"}
Sep 16 21:55:33.050684 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:33.050283    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6" event={"ID":"f2108207-6b4e-4e1a-87b9-de4e665af470","Type":"ContainerDied","Data":"70f97e01e7e2fd52939d462de1bca5a9c38b731635b51f93d53baf5086be3bca"}
Sep 16 21:55:33.050684 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:33.050306    2567 scope.go:117] "RemoveContainer" containerID="4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273"
Sep 16 21:55:33.060348 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:33.060195    2567 scope.go:117] "RemoveContainer" containerID="4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273"
Sep 16 21:55:33.060513 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:55:33.060490    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273\": container with ID starting with 4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273 not found: ID does not exist" containerID="4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273"
Sep 16 21:55:33.060579 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:33.060524    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273"} err="failed to get container status \"4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273\": rpc error: code = NotFound desc = could not find container \"4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273\": container with ID starting with 4201c65794d98968124d9325614f7996817d74fcc728d61b4919e18848202273 not found: ID does not exist"
Sep 16 21:55:33.070623 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:33.070604    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"]
Sep 16 21:55:33.074155 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:33.074137    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1b68db-75d5b4d7f8-knpj6"]
Sep 16 21:55:33.108115 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:33.108089    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3c80c73-257a-4747-93c9-86fbeb88d781" path="/var/lib/kubelet/pods/c3c80c73-257a-4747-93c9-86fbeb88d781/volumes"
Sep 16 21:55:33.108462 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:33.108446    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2108207-6b4e-4e1a-87b9-de4e665af470" path="/var/lib/kubelet/pods/f2108207-6b4e-4e1a-87b9-de4e665af470/volumes"
Sep 16 21:55:34.575804 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:34.575765    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:55:41.585552 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:41.585517    2567 scope.go:117] "RemoveContainer" containerID="1066ef53cf07d8ad3df3d23290b236100a40d6b238b94bf0199ecf3e9e3d9d1d"
Sep 16 21:55:43.174981 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:55:43.174948    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:57:06.074632 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:57:06.074550    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:57:31.573917 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:57:31.573879    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:57:39.583846 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:57:39.583806    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:57:47.682857 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:57:47.682814    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:58:09.788983 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:09.788939    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:58:15.798469 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:15.798428    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:58:17.273702 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:17.273665    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"]
Sep 16 21:58:17.274097 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:17.273888    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm" podUID="82aba2fe-2314-44ec-a8a8-404b6db3af87" containerName="manager" containerID="cri-o://7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a" gracePeriod=10
Sep 16 21:58:18.713508 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.713485    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:58:18.760537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.760507    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82aba2fe-2314-44ec-a8a8-404b6db3af87-extensions-socket-volume\") pod \"82aba2fe-2314-44ec-a8a8-404b6db3af87\" (UID: \"82aba2fe-2314-44ec-a8a8-404b6db3af87\") "
Sep 16 21:58:18.760659 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.760623    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb5tc\" (UniqueName: \"kubernetes.io/projected/82aba2fe-2314-44ec-a8a8-404b6db3af87-kube-api-access-jb5tc\") pod \"82aba2fe-2314-44ec-a8a8-404b6db3af87\" (UID: \"82aba2fe-2314-44ec-a8a8-404b6db3af87\") "
Sep 16 21:58:18.760918 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.760891    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82aba2fe-2314-44ec-a8a8-404b6db3af87-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "82aba2fe-2314-44ec-a8a8-404b6db3af87" (UID: "82aba2fe-2314-44ec-a8a8-404b6db3af87"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:58:18.762569 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.762547    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82aba2fe-2314-44ec-a8a8-404b6db3af87-kube-api-access-jb5tc" (OuterVolumeSpecName: "kube-api-access-jb5tc") pod "82aba2fe-2314-44ec-a8a8-404b6db3af87" (UID: "82aba2fe-2314-44ec-a8a8-404b6db3af87"). InnerVolumeSpecName "kube-api-access-jb5tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:58:18.862273 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.862201    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb5tc\" (UniqueName: \"kubernetes.io/projected/82aba2fe-2314-44ec-a8a8-404b6db3af87-kube-api-access-jb5tc\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:58:18.862273 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.862225    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82aba2fe-2314-44ec-a8a8-404b6db3af87-extensions-socket-volume\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 21:58:18.979485 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.979449    2567 generic.go:356] "Generic (PLEG): container finished" podID="82aba2fe-2314-44ec-a8a8-404b6db3af87" containerID="7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a" exitCode=0
Sep 16 21:58:18.979628 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.979496    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"
Sep 16 21:58:18.979628 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.979513    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm" event={"ID":"82aba2fe-2314-44ec-a8a8-404b6db3af87","Type":"ContainerDied","Data":"7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a"}
Sep 16 21:58:18.979628 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.979581    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm" event={"ID":"82aba2fe-2314-44ec-a8a8-404b6db3af87","Type":"ContainerDied","Data":"e92c5bbbd5d540dab754f9dc90d34103811883d2af696d0efc48ef34ff8eed8c"}
Sep 16 21:58:18.979628 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.979602    2567 scope.go:117] "RemoveContainer" containerID="7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a"
Sep 16 21:58:18.989016 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.988999    2567 scope.go:117] "RemoveContainer" containerID="bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb"
Sep 16 21:58:18.997061 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.997039    2567 scope.go:117] "RemoveContainer" containerID="7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a"
Sep 16 21:58:18.997291 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:58:18.997272    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a\": container with ID starting with 7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a not found: ID does not exist" containerID="7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a"
Sep 16 21:58:18.997374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.997303    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a"} err="failed to get container status \"7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a\": rpc error: code = NotFound desc = could not find container \"7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a\": container with ID starting with 7660ec35de2b072470bbe94d45d07f5a69ab9ec0de6dfa3d729c6a295e8db38a not found: ID does not exist"
Sep 16 21:58:18.997374 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.997325    2567 scope.go:117] "RemoveContainer" containerID="bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb"
Sep 16 21:58:18.997884 ip-10-0-140-93 kubenswrapper[2567]: E0916 21:58:18.997861    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb\": container with ID starting with bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb not found: ID does not exist" containerID="bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb"
Sep 16 21:58:18.997946 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:18.997891    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb"} err="failed to get container status \"bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb\": rpc error: code = NotFound desc = could not find container \"bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb\": container with ID starting with bdb3ac8886b83c7bdb713ba6627cbacee0563aff50c8f5af9009b4786a4d14cb not found: ID does not exist"
Sep 16 21:58:19.000445 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:19.000423    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"]
Sep 16 21:58:19.004143 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:19.004124    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tnvgm"]
Sep 16 21:58:19.108067 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:58:19.108039    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82aba2fe-2314-44ec-a8a8-404b6db3af87" path="/var/lib/kubelet/pods/82aba2fe-2314-44ec-a8a8-404b6db3af87/volumes"
Sep 16 21:59:17.976901 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.976859    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"]
Sep 16 21:59:17.977537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977415    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82aba2fe-2314-44ec-a8a8-404b6db3af87" containerName="manager"
Sep 16 21:59:17.977537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977438    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="82aba2fe-2314-44ec-a8a8-404b6db3af87" containerName="manager"
Sep 16 21:59:17.977537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977454    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cace92e0-5248-4fbe-84b8-534efbf590d8" containerName="payload-pre-processing"
Sep 16 21:59:17.977537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977463    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cace92e0-5248-4fbe-84b8-534efbf590d8" containerName="payload-pre-processing"
Sep 16 21:59:17.977537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977474    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f00088-7730-4288-b556-fe91c9436ee9" containerName="payload-processing"
Sep 16 21:59:17.977537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977484    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f00088-7730-4288-b556-fe91c9436ee9" containerName="payload-processing"
Sep 16 21:59:17.977537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977496    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dce157d-91a4-4d82-9505-7cee2a16f2ce" containerName="revoke-keys"
Sep 16 21:59:17.977537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977505    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dce157d-91a4-4d82-9505-7cee2a16f2ce" containerName="revoke-keys"
Sep 16 21:59:17.977537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977516    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2108207-6b4e-4e1a-87b9-de4e665af470" containerName="payload-pre-processing"
Sep 16 21:59:17.977537 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977525    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2108207-6b4e-4e1a-87b9-de4e665af470" containerName="payload-pre-processing"
Sep 16 21:59:17.978054 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977637    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dce157d-91a4-4d82-9505-7cee2a16f2ce" containerName="revoke-keys"
Sep 16 21:59:17.978054 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977650    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6f00088-7730-4288-b556-fe91c9436ee9" containerName="payload-processing"
Sep 16 21:59:17.978054 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977665    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="82aba2fe-2314-44ec-a8a8-404b6db3af87" containerName="manager"
Sep 16 21:59:17.978054 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977676    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2108207-6b4e-4e1a-87b9-de4e665af470" containerName="payload-pre-processing"
Sep 16 21:59:17.978054 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977686    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="82aba2fe-2314-44ec-a8a8-404b6db3af87" containerName="manager"
Sep 16 21:59:17.978054 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.977694    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cace92e0-5248-4fbe-84b8-534efbf590d8" containerName="payload-pre-processing"
Sep 16 21:59:17.980743 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.980722    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"
Sep 16 21:59:17.982978 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.982955    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kgfv7\""
Sep 16 21:59:17.994446 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:17.994425    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"]
Sep 16 21:59:18.018112 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:18.018084    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6chv\" (UniqueName: \"kubernetes.io/projected/bfdf0737-3e2b-4874-86ca-048529b7f8a7-kube-api-access-n6chv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5nm2x\" (UID: \"bfdf0737-3e2b-4874-86ca-048529b7f8a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"
Sep 16 21:59:18.018226 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:18.018117    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfdf0737-3e2b-4874-86ca-048529b7f8a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5nm2x\" (UID: \"bfdf0737-3e2b-4874-86ca-048529b7f8a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"
Sep 16 21:59:18.118652 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:18.118629    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6chv\" (UniqueName: \"kubernetes.io/projected/bfdf0737-3e2b-4874-86ca-048529b7f8a7-kube-api-access-n6chv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5nm2x\" (UID: \"bfdf0737-3e2b-4874-86ca-048529b7f8a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"
Sep 16 21:59:18.118738 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:18.118658    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfdf0737-3e2b-4874-86ca-048529b7f8a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5nm2x\" (UID: \"bfdf0737-3e2b-4874-86ca-048529b7f8a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"
Sep 16 21:59:18.118983 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:18.118967    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfdf0737-3e2b-4874-86ca-048529b7f8a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5nm2x\" (UID: \"bfdf0737-3e2b-4874-86ca-048529b7f8a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"
Sep 16 21:59:18.127529 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:18.127506    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6chv\" (UniqueName: \"kubernetes.io/projected/bfdf0737-3e2b-4874-86ca-048529b7f8a7-kube-api-access-n6chv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5nm2x\" (UID: \"bfdf0737-3e2b-4874-86ca-048529b7f8a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"
Sep 16 21:59:18.290561 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:18.290477    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"
Sep 16 21:59:18.572309 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:18.572283    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"]
Sep 16 21:59:18.574956 ip-10-0-140-93 kubenswrapper[2567]: W0916 21:59:18.574925    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdf0737_3e2b_4874_86ca_048529b7f8a7.slice/crio-a69b4acdce8c351ae1375f134fc5710338d4f31f466bd312c04d7618b5fd563e WatchSource:0}: Error finding container a69b4acdce8c351ae1375f134fc5710338d4f31f466bd312c04d7618b5fd563e: Status 404 returned error can't find the container with id a69b4acdce8c351ae1375f134fc5710338d4f31f466bd312c04d7618b5fd563e
Sep 16 21:59:19.314323 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:19.314271    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x" event={"ID":"bfdf0737-3e2b-4874-86ca-048529b7f8a7","Type":"ContainerStarted","Data":"3ca7a8770d2603881e537bbe3cf63cde385691829bf5fcced84c086ae3d3a65f"}
Sep 16 21:59:19.314797 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:19.314348    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x" event={"ID":"bfdf0737-3e2b-4874-86ca-048529b7f8a7","Type":"ContainerStarted","Data":"a69b4acdce8c351ae1375f134fc5710338d4f31f466bd312c04d7618b5fd563e"}
Sep 16 21:59:19.314797 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:19.314501    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"
Sep 16 21:59:19.335865 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:19.335813    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x" podStartSLOduration=2.33579291 podStartE2EDuration="2.33579291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:59:19.335677777 +0000 UTC m=+2198.721360881" watchObservedRunningTime="2026-09-16 21:59:19.33579291 +0000 UTC m=+2198.721475995"
Sep 16 21:59:30.322273 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:30.322239    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5nm2x"
Sep 16 21:59:42.184286 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:42.184248    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 21:59:51.376699 ip-10-0-140-93 kubenswrapper[2567]: I0916 21:59:51.376657    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 22:00:02.187301 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:02.187263    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 22:00:11.078743 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.078659    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"]
Sep 16 22:00:11.079226 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.079205    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82aba2fe-2314-44ec-a8a8-404b6db3af87" containerName="manager"
Sep 16 22:00:11.079308 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.079228    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="82aba2fe-2314-44ec-a8a8-404b6db3af87" containerName="manager"
Sep 16 22:00:11.082634 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.082614    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.084955 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.084932    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-44cf9098-kserve-self-signed-certs\""
Sep 16 22:00:11.091909 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.091886    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"]
Sep 16 22:00:11.155017 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.154983    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkd9l\" (UniqueName: \"kubernetes.io/projected/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-kube-api-access-dkd9l\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.155165 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.155064    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tmp-dir\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.155165 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.155082    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-model-cache\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.155165 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.155098    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-dshm\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.155165 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.155127    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tls-certs\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.155165 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.155141    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-home\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.255609 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.255569    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tmp-dir\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.255609 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.255609    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-model-cache\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.255856 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.255627    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-dshm\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.255856 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.255768    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tls-certs\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.255856 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.255816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-home\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.256022 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.255872    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkd9l\" (UniqueName: \"kubernetes.io/projected/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-kube-api-access-dkd9l\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.256086 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.256061    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tmp-dir\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.256144 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.256111    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-model-cache\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.256144 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.256117    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-home\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.257802 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.257778    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-dshm\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.258140 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.258121    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tls-certs\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.264079 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.264056    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkd9l\" (UniqueName: \"kubernetes.io/projected/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-kube-api-access-dkd9l\") pod \"e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") " pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.392276 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.392209    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:11.668378 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:11.668318    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"]
Sep 16 22:00:11.671060 ip-10-0-140-93 kubenswrapper[2567]: W0916 22:00:11.671030    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9050f0a6_2a90_4350_9a46_cb0082f5ad6d.slice/crio-f2c50be99f7380ea54c081745b3e9a33e92d48b248dac9b8920d115d30c63b5d WatchSource:0}: Error finding container f2c50be99f7380ea54c081745b3e9a33e92d48b248dac9b8920d115d30c63b5d: Status 404 returned error can't find the container with id f2c50be99f7380ea54c081745b3e9a33e92d48b248dac9b8920d115d30c63b5d
Sep 16 22:00:12.618027 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:12.617992    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6" event={"ID":"9050f0a6-2a90-4350-9a46-cb0082f5ad6d","Type":"ContainerStarted","Data":"652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b"}
Sep 16 22:00:12.618526 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:12.618042    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6" event={"ID":"9050f0a6-2a90-4350-9a46-cb0082f5ad6d","Type":"ContainerStarted","Data":"f2c50be99f7380ea54c081745b3e9a33e92d48b248dac9b8920d115d30c63b5d"}
Sep 16 22:00:12.635571 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:12.635519    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6" podStartSLOduration=1.635502647 podStartE2EDuration="1.635502647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 22:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 22:00:12.633250943 +0000 UTC m=+2252.018934029" watchObservedRunningTime="2026-09-16 22:00:12.635502647 +0000 UTC m=+2252.021185731"
Sep 16 22:00:13.882915 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:13.882877    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 22:00:21.393275 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:21.393238    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:21.393676 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:21.393311    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:21.405679 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:21.405659    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:21.681041 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:21.680962    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:29.070600 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:29.070568    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"]
Sep 16 22:00:31.083222 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:31.083182    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 22:00:38.383692 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:38.383652    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 22:00:42.879500 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:42.879461    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 22:00:44.091796 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.091742    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6" podUID="9050f0a6-2a90-4350-9a46-cb0082f5ad6d" containerName="main" containerID="cri-o://652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b" gracePeriod=45
Sep 16 22:00:44.347053 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.346996    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:44.451843 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.451803    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-home\") pod \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") "
Sep 16 22:00:44.452042 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.451875    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-model-cache\") pod \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") "
Sep 16 22:00:44.452042 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.451923    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tmp-dir\") pod \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") "
Sep 16 22:00:44.452042 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.451987    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tls-certs\") pod \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") "
Sep 16 22:00:44.452042 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.452023    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkd9l\" (UniqueName: \"kubernetes.io/projected/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-kube-api-access-dkd9l\") pod \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") "
Sep 16 22:00:44.452255 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.452053    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-dshm\") pod \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\" (UID: \"9050f0a6-2a90-4350-9a46-cb0082f5ad6d\") "
Sep 16 22:00:44.452255 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.452093    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-home" (OuterVolumeSpecName: "home") pod "9050f0a6-2a90-4350-9a46-cb0082f5ad6d" (UID: "9050f0a6-2a90-4350-9a46-cb0082f5ad6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 22:00:44.452255 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.452189    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9050f0a6-2a90-4350-9a46-cb0082f5ad6d" (UID: "9050f0a6-2a90-4350-9a46-cb0082f5ad6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 22:00:44.452255 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.452217    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9050f0a6-2a90-4350-9a46-cb0082f5ad6d" (UID: "9050f0a6-2a90-4350-9a46-cb0082f5ad6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 22:00:44.452387 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.452354    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tmp-dir\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 22:00:44.452387 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.452369    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-home\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 22:00:44.452387 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.452380    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-model-cache\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 22:00:44.454381 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.454345    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-dshm" (OuterVolumeSpecName: "dshm") pod "9050f0a6-2a90-4350-9a46-cb0082f5ad6d" (UID: "9050f0a6-2a90-4350-9a46-cb0082f5ad6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 22:00:44.454381 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.454360    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-kube-api-access-dkd9l" (OuterVolumeSpecName: "kube-api-access-dkd9l") pod "9050f0a6-2a90-4350-9a46-cb0082f5ad6d" (UID: "9050f0a6-2a90-4350-9a46-cb0082f5ad6d"). InnerVolumeSpecName "kube-api-access-dkd9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 22:00:44.454559 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.454466    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9050f0a6-2a90-4350-9a46-cb0082f5ad6d" (UID: "9050f0a6-2a90-4350-9a46-cb0082f5ad6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 22:00:44.553131 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.553087    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkd9l\" (UniqueName: \"kubernetes.io/projected/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-kube-api-access-dkd9l\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 22:00:44.553131 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.553130    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-dshm\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 22:00:44.553334 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.553142    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9050f0a6-2a90-4350-9a46-cb0082f5ad6d-tls-certs\") on node \"ip-10-0-140-93.ec2.internal\" DevicePath \"\""
Sep 16 22:00:44.803599 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.803557    2567 generic.go:356] "Generic (PLEG): container finished" podID="9050f0a6-2a90-4350-9a46-cb0082f5ad6d" containerID="652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b" exitCode=0
Sep 16 22:00:44.803599 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.803600    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"
Sep 16 22:00:44.803869 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.803641    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6" event={"ID":"9050f0a6-2a90-4350-9a46-cb0082f5ad6d","Type":"ContainerDied","Data":"652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b"}
Sep 16 22:00:44.803869 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.803683    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6" event={"ID":"9050f0a6-2a90-4350-9a46-cb0082f5ad6d","Type":"ContainerDied","Data":"f2c50be99f7380ea54c081745b3e9a33e92d48b248dac9b8920d115d30c63b5d"}
Sep 16 22:00:44.803869 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.803703    2567 scope.go:117] "RemoveContainer" containerID="652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b"
Sep 16 22:00:44.813299 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.813284    2567 scope.go:117] "RemoveContainer" containerID="652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b"
Sep 16 22:00:44.813585 ip-10-0-140-93 kubenswrapper[2567]: E0916 22:00:44.813567    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b\": container with ID starting with 652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b not found: ID does not exist" containerID="652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b"
Sep 16 22:00:44.813661 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.813598    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b"} err="failed to get container status \"652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b\": rpc error: code = NotFound desc = could not find container \"652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b\": container with ID starting with 652e361d3852b9c733b3e4683636ac0abdf628dd8702fd3540ed52975169fb9b not found: ID does not exist"
Sep 16 22:00:44.825010 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.824987    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"]
Sep 16 22:00:44.832074 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:44.832053    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-44cf9098-kserve-7674c84dd7-rj5z6"]
Sep 16 22:00:45.108569 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:45.108487    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9050f0a6-2a90-4350-9a46-cb0082f5ad6d" path="/var/lib/kubelet/pods/9050f0a6-2a90-4350-9a46-cb0082f5ad6d/volumes"
Sep 16 22:00:50.776460 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.776417    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 22:00:50.897038 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.896999    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"]
Sep 16 22:00:50.897453 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.897439    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9050f0a6-2a90-4350-9a46-cb0082f5ad6d" containerName="main"
Sep 16 22:00:50.897526 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.897455    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9050f0a6-2a90-4350-9a46-cb0082f5ad6d" containerName="main"
Sep 16 22:00:50.897526 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.897515    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9050f0a6-2a90-4350-9a46-cb0082f5ad6d" containerName="main"
Sep 16 22:00:50.900679 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.900660    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:50.903412 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.903382    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 22:00:50.903542 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.903382    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 22:00:50.903542 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.903440    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lnht4\""
Sep 16 22:00:50.903542 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.903461    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 22:00:50.903542 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.903509    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 22:00:50.909901 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:50.909879    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"]
Sep 16 22:00:51.000344 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.000310    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbj55\" (UniqueName: \"kubernetes.io/projected/7758a19c-63fd-48e3-bdb7-5c7035a325c0-kube-api-access-mbj55\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.000546 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.000350    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7758a19c-63fd-48e3-bdb7-5c7035a325c0-openshift-service-ca\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.000546 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.000424    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7758a19c-63fd-48e3-bdb7-5c7035a325c0-maas-api-tls\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.000546 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.000456    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7758a19c-63fd-48e3-bdb7-5c7035a325c0-metrics-certs\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.101126 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.101043    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbj55\" (UniqueName: \"kubernetes.io/projected/7758a19c-63fd-48e3-bdb7-5c7035a325c0-kube-api-access-mbj55\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.101126 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.101088    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7758a19c-63fd-48e3-bdb7-5c7035a325c0-openshift-service-ca\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.101293 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.101132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7758a19c-63fd-48e3-bdb7-5c7035a325c0-maas-api-tls\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.101293 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.101165    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7758a19c-63fd-48e3-bdb7-5c7035a325c0-metrics-certs\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.101957 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.101933    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7758a19c-63fd-48e3-bdb7-5c7035a325c0-openshift-service-ca\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.104330 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.104301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7758a19c-63fd-48e3-bdb7-5c7035a325c0-maas-api-tls\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.104330 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.104317    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7758a19c-63fd-48e3-bdb7-5c7035a325c0-metrics-certs\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.109180 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.109156    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbj55\" (UniqueName: \"kubernetes.io/projected/7758a19c-63fd-48e3-bdb7-5c7035a325c0-kube-api-access-mbj55\") pod \"maas-api-6789c486f4-l5nk8\" (UID: \"7758a19c-63fd-48e3-bdb7-5c7035a325c0\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.212274 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.212232    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:51.495270 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.495245    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"]
Sep 16 22:00:51.497577 ip-10-0-140-93 kubenswrapper[2567]: W0916 22:00:51.497549    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7758a19c_63fd_48e3_bdb7_5c7035a325c0.slice/crio-55fe1fd33b49e3320b9a52dd2255455ea5afccb3689e4d70f3cf2beb9def8f33 WatchSource:0}: Error finding container 55fe1fd33b49e3320b9a52dd2255455ea5afccb3689e4d70f3cf2beb9def8f33: Status 404 returned error can't find the container with id 55fe1fd33b49e3320b9a52dd2255455ea5afccb3689e4d70f3cf2beb9def8f33
Sep 16 22:00:51.498982 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.498959    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 22:00:51.846906 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:51.846816    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8" event={"ID":"7758a19c-63fd-48e3-bdb7-5c7035a325c0","Type":"ContainerStarted","Data":"55fe1fd33b49e3320b9a52dd2255455ea5afccb3689e4d70f3cf2beb9def8f33"}
Sep 16 22:00:55.872894 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:55.872857    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8" event={"ID":"7758a19c-63fd-48e3-bdb7-5c7035a325c0","Type":"ContainerStarted","Data":"2ecefeb618e9b8c6b73d781715ed6eac6c9cb87ab0cc53ff1d6a7972c5a2b8a6"}
Sep 16 22:00:55.873321 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:55.872988    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:00:55.892075 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:55.892029    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8" podStartSLOduration=2.272361079 podStartE2EDuration="5.89201621s" totalImagesPullingTime="3.619655131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 22:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 22:00:55.888922449 +0000 UTC m=+2295.274605547" watchObservedRunningTime="2026-09-16 22:00:55.89201621 +0000 UTC m=+2295.277699311"
Sep 16 22:00:57.373542 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:00:57.373505    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 22:01:01.883485 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:01.883453    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6789c486f4-l5nk8"
Sep 16 22:01:03.077308 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:03.077268    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 22:01:09.873468 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:09.873433    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zljw4"]
Sep 16 22:01:18.124873 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:18.124846    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wvrjt_272ac6f7-a92e-45c8-869d-c6a9b4bc9682/manager/0.log"
Sep 16 22:01:18.248246 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:18.248212    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tbpfn_da9e2bd5-c84e-494f-815e-135e7d3dc26b/manager/0.log"
Sep 16 22:01:18.816525 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:18.816497    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-dpdnf_611804fc-3070-49eb-b152-b2459ec900cb/copy-manifests/0.log"
Sep 16 22:01:18.823135 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:18.823105    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-dpdnf_611804fc-3070-49eb-b152-b2459ec900cb/manager/0.log"
Sep 16 22:01:19.379332 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:19.379298    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6789c486f4-l5nk8_7758a19c-63fd-48e3-bdb7-5c7035a325c0/maas-api/0.log"
Sep 16 22:01:20.664792 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:20.664753    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-fpstz_e394fbd6-59c8-4157-8bb5-0fc3e40a70ac/manager/0.log"
Sep 16 22:01:20.899891 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:20.899856    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-5nm2x_bfdf0737-3e2b-4874-86ca-048529b7f8a7/manager/0.log"
Sep 16 22:01:21.002190 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:21.002165    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-zljw4_2bacf858-59b2-48be-81e4-b2eb98d0b66f/limitador/0.log"
Sep 16 22:01:21.125541 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:21.125504    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-48k77_1557051d-7d49-44a5-8811-664cd89a72a2/manager/0.log"
Sep 16 22:01:21.569281 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:21.569245    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xcdvq_67d01f03-a6c2-4075-8999-9a667273878d/discovery/0.log"
Sep 16 22:01:21.676389 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:21.676366    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f65bc968c-vt5vn_ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd/kube-auth-proxy/0.log"
Sep 16 22:01:22.126370 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:22.126340    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-px5dq_2babf7f2-b5ca-4aa7-af2a-bd60178bac8c/payload-processing/0.log"
Sep 16 22:01:22.903317 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:22.903282    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-p4qwx_af2d1205-80da-4300-8db9-f5aa457d74e8/main/0.log"
Sep 16 22:01:32.545258 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:32.545216    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j92fr_150f4568-938b-47df-a346-09c3fd2e5dac/global-pull-secret-syncer/0.log"
Sep 16 22:01:32.674018 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:32.673983    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-khthp_cba9022f-9608-47da-8766-4fd3f31909d4/konnectivity-agent/0.log"
Sep 16 22:01:32.774490 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:32.774453    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-93.ec2.internal_6bab6030418b48a95c5e7134d86b1cea/haproxy/0.log"
Sep 16 22:01:37.958054 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:37.957971    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-fpstz_e394fbd6-59c8-4157-8bb5-0fc3e40a70ac/manager/0.log"
Sep 16 22:01:38.026210 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:38.026180    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-5nm2x_bfdf0737-3e2b-4874-86ca-048529b7f8a7/manager/0.log"
Sep 16 22:01:38.045363 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:38.045332    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-zljw4_2bacf858-59b2-48be-81e4-b2eb98d0b66f/limitador/0.log"
Sep 16 22:01:38.121831 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:38.121800    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-48k77_1557051d-7d49-44a5-8811-664cd89a72a2/manager/0.log"
Sep 16 22:01:39.645372 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.645331    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_652abb68-3710-4f10-a7e0-7e49bef35826/alertmanager/0.log"
Sep 16 22:01:39.676622 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.676597    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_652abb68-3710-4f10-a7e0-7e49bef35826/config-reloader/0.log"
Sep 16 22:01:39.701973 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.701949    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_652abb68-3710-4f10-a7e0-7e49bef35826/kube-rbac-proxy-web/0.log"
Sep 16 22:01:39.728546 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.728524    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_652abb68-3710-4f10-a7e0-7e49bef35826/kube-rbac-proxy/0.log"
Sep 16 22:01:39.751567 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.751547    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_652abb68-3710-4f10-a7e0-7e49bef35826/kube-rbac-proxy-metric/0.log"
Sep 16 22:01:39.772570 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.772552    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_652abb68-3710-4f10-a7e0-7e49bef35826/prom-label-proxy/0.log"
Sep 16 22:01:39.792843 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.792820    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_652abb68-3710-4f10-a7e0-7e49bef35826/init-config-reloader/0.log"
Sep 16 22:01:39.890930 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.890910    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-kftsz_aa0bee59-8a71-49a2-bd47-837d1153c745/kube-state-metrics/0.log"
Sep 16 22:01:39.910048 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.910027    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-kftsz_aa0bee59-8a71-49a2-bd47-837d1153c745/kube-rbac-proxy-main/0.log"
Sep 16 22:01:39.936518 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.936494    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-kftsz_aa0bee59-8a71-49a2-bd47-837d1153c745/kube-rbac-proxy-self/0.log"
Sep 16 22:01:39.967713 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.967687    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6f848466bd-gqxfc_9179cf9f-ac35-4f83-94d1-2c170833c6cf/metrics-server/0.log"
Sep 16 22:01:39.996301 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:39.996280    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-t5fcf_2612272b-ccf5-4e52-b710-afb85dff8c7a/monitoring-plugin/0.log"
Sep 16 22:01:40.145486 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:40.145422    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qwrmj_474831f6-ad89-4837-b240-e1db1f74c15e/node-exporter/0.log"
Sep 16 22:01:40.168692 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:40.168673    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qwrmj_474831f6-ad89-4837-b240-e1db1f74c15e/kube-rbac-proxy/0.log"
Sep 16 22:01:40.190068 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:40.190053    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qwrmj_474831f6-ad89-4837-b240-e1db1f74c15e/init-textfile/0.log"
Sep 16 22:01:40.589524 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:40.589488    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-85f7w_f62ceb82-fb55-410e-8c06-9798e256d099/prometheus-operator/0.log"
Sep 16 22:01:40.610383 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:40.610357    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-85f7w_f62ceb82-fb55-410e-8c06-9798e256d099/kube-rbac-proxy/0.log"
Sep 16 22:01:40.767103 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:40.767075    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9bdff54c7-w748q_b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e/thanos-query/0.log"
Sep 16 22:01:40.796127 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:40.796103    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9bdff54c7-w748q_b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e/kube-rbac-proxy-web/0.log"
Sep 16 22:01:40.815657 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:40.815631    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9bdff54c7-w748q_b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e/kube-rbac-proxy/0.log"
Sep 16 22:01:40.834190 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:40.834171    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9bdff54c7-w748q_b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e/prom-label-proxy/0.log"
Sep 16 22:01:40.852962 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:40.852908    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9bdff54c7-w748q_b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e/kube-rbac-proxy-rules/0.log"
Sep 16 22:01:40.874540 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:40.874519    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9bdff54c7-w748q_b9cc2ce0-5b8b-41a0-a670-a9a81d0f7f9e/kube-rbac-proxy-metrics/0.log"
Sep 16 22:01:41.681407 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:41.681376    2567 scope.go:117] "RemoveContainer" containerID="9d489b30d5c977b69eae1b579fdee321f0d3cca66b54b17d378dc1854fb1cfe4"
Sep 16 22:01:41.690080 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:41.690058    2567 scope.go:117] "RemoveContainer" containerID="0ab6b9b191dc5c94e07dc8f9f586c7d2e18afaf11246479509d3d6f2dffd94af"
Sep 16 22:01:42.983137 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:42.983102    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"]
Sep 16 22:01:42.988034 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:42.988012    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:42.990111 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:42.990090    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfn8w\"/\"openshift-service-ca.crt\""
Sep 16 22:01:42.990222 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:42.990206    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfn8w\"/\"kube-root-ca.crt\""
Sep 16 22:01:42.990863 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:42.990825    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pfn8w\"/\"default-dockercfg-zfj9l\""
Sep 16 22:01:42.996745 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:42.996723    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"]
Sep 16 22:01:43.140900 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.140866    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-podres\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.140900 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.140899    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l7gr\" (UniqueName: \"kubernetes.io/projected/7e5bf138-2109-4025-a0dc-6cb18f5af4de-kube-api-access-5l7gr\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.141124 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.140922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-sys\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.141124 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.141018    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-lib-modules\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.141124 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.141064    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-proc\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.242287 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.242202    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-podres\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.242287 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.242242    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l7gr\" (UniqueName: \"kubernetes.io/projected/7e5bf138-2109-4025-a0dc-6cb18f5af4de-kube-api-access-5l7gr\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.242287 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.242267    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-sys\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.242570 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.242303    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-lib-modules\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.242570 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.242326    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-proc\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.242570 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.242421    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-podres\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.242570 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.242433    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-proc\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.242570 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.242433    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-sys\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.242570 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.242472    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e5bf138-2109-4025-a0dc-6cb18f5af4de-lib-modules\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.249797 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.249777    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l7gr\" (UniqueName: \"kubernetes.io/projected/7e5bf138-2109-4025-a0dc-6cb18f5af4de-kube-api-access-5l7gr\") pod \"perf-node-gather-daemonset-4lk2w\" (UID: \"7e5bf138-2109-4025-a0dc-6cb18f5af4de\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.298664 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.298642    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:43.567243 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:43.567213    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"]
Sep 16 22:01:43.569145 ip-10-0-140-93 kubenswrapper[2567]: W0916 22:01:43.569115    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7e5bf138_2109_4025_a0dc_6cb18f5af4de.slice/crio-1c38148aec210c98521b889c474ca17467602bd564c2c19cb4aec8e36163c2a1 WatchSource:0}: Error finding container 1c38148aec210c98521b889c474ca17467602bd564c2c19cb4aec8e36163c2a1: Status 404 returned error can't find the container with id 1c38148aec210c98521b889c474ca17467602bd564c2c19cb4aec8e36163c2a1
Sep 16 22:01:44.143099 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:44.143067    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w" event={"ID":"7e5bf138-2109-4025-a0dc-6cb18f5af4de","Type":"ContainerStarted","Data":"f0f2083b09a012a632b6685bf8b9e66b15af19dc607943689de896a6da940376"}
Sep 16 22:01:44.143099 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:44.143117    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w" event={"ID":"7e5bf138-2109-4025-a0dc-6cb18f5af4de","Type":"ContainerStarted","Data":"1c38148aec210c98521b889c474ca17467602bd564c2c19cb4aec8e36163c2a1"}
Sep 16 22:01:44.143572 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:44.143213    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:44.158930 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:44.158886    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w" podStartSLOduration=2.158873169 podStartE2EDuration="2.158873169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 22:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 22:01:44.157383726 +0000 UTC m=+2343.543066822" watchObservedRunningTime="2026-09-16 22:01:44.158873169 +0000 UTC m=+2343.544556252"
Sep 16 22:01:45.200546 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:45.200506    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-98sxj_42fd9e51-d6c3-4173-a89a-32dc02b95364/dns/0.log"
Sep 16 22:01:45.220091 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:45.220068    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-98sxj_42fd9e51-d6c3-4173-a89a-32dc02b95364/kube-rbac-proxy/0.log"
Sep 16 22:01:45.311653 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:45.311628    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j4dhm_7a5797dc-757d-4e24-8cd8-e37626fe3ff4/dns-node-resolver/0.log"
Sep 16 22:01:45.918535 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:45.918505    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pgpsw_7fa8e414-a609-4b5e-9343-1cda7ab8b199/node-ca/0.log"
Sep 16 22:01:47.624044 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:47.624001    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xcdvq_67d01f03-a6c2-4075-8999-9a667273878d/discovery/0.log"
Sep 16 22:01:47.645142 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:47.645119    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f65bc968c-vt5vn_ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd/kube-auth-proxy/0.log"
Sep 16 22:01:47.779537 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:47.779504    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-px5dq_2babf7f2-b5ca-4aa7-af2a-bd60178bac8c/payload-processing/0.log"
Sep 16 22:01:48.698956 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:48.698923    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6n7r5_1843cef4-b35a-44e8-9106-c2e609602879/serve-healthcheck-canary/0.log"
Sep 16 22:01:49.358271 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:49.358243    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pft6k_61943deb-41b6-4b74-8eb3-e9812d5a0797/kube-rbac-proxy/0.log"
Sep 16 22:01:49.376269 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:49.376240    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pft6k_61943deb-41b6-4b74-8eb3-e9812d5a0797/exporter/0.log"
Sep 16 22:01:49.399056 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:49.399034    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pft6k_61943deb-41b6-4b74-8eb3-e9812d5a0797/extractor/0.log"
Sep 16 22:01:50.158494 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:50.158462    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-4lk2w"
Sep 16 22:01:51.637176 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:51.637153    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wvrjt_272ac6f7-a92e-45c8-869d-c6a9b4bc9682/manager/0.log"
Sep 16 22:01:52.127142 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:52.127058    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tbpfn_da9e2bd5-c84e-494f-815e-135e7d3dc26b/manager/0.log"
Sep 16 22:01:52.408285 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:52.408203    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-dpdnf_611804fc-3070-49eb-b152-b2459ec900cb/manager/0.log"
Sep 16 22:01:52.432765 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:52.432732    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-dpdnf_611804fc-3070-49eb-b152-b2459ec900cb/copy-manifests/0.log"
Sep 16 22:01:53.883264 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:53.883235    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wn77f_cb4bf756-e00e-45af-9659-9233f59f9ee6/openshift-lws-operator/0.log"
Sep 16 22:01:54.209877 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:54.209848    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xcdvq_67d01f03-a6c2-4075-8999-9a667273878d/discovery/0.log"
Sep 16 22:01:54.222931 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:54.222904    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f65bc968c-vt5vn_ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd/kube-auth-proxy/0.log"
Sep 16 22:01:54.326852 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:54.326826    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-px5dq_2babf7f2-b5ca-4aa7-af2a-bd60178bac8c/payload-processing/0.log"
Sep 16 22:01:56.562040 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:56.562000    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xcdvq_67d01f03-a6c2-4075-8999-9a667273878d/discovery/0.log"
Sep 16 22:01:56.586645 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:56.586623    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f65bc968c-vt5vn_ae4ca6d9-cadd-4f8a-ac97-300d78b63ecd/kube-auth-proxy/0.log"
Sep 16 22:01:56.699372 ip-10-0-140-93 kubenswrapper[2567]: I0916 22:01:56.699345    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-px5dq_2babf7f2-b5ca-4aa7-af2a-bd60178bac8c/payload-processing/0.log"