Sep 16 18:25:48.068530 ip-10-0-141-16 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:25:48.068545 ip-10-0-141-16 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:25:48.068555 ip-10-0-141-16 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:25:48.068780 ip-10-0-141-16 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:25:59.135919 ip-10-0-141-16 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:25:59.135934 ip-10-0-141-16 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 83744419043f4cb4b8df4e1d52f71ff7 --
Sep 16 18:27:37.170599 ip-10-0-141-16 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:27:37.589421 ip-10-0-141-16 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:27:37.589421 ip-10-0-141-16 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:27:37.589421 ip-10-0-141-16 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:27:37.589421 ip-10-0-141-16 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:27:37.589421 ip-10-0-141-16 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:27:37.590574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.590487    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:27:37.595714 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595698    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:27:37.595714 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595714    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595718    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595721    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595724    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595727    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595729    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595732    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595735    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595738    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595741    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595743    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595746    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595750    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595752    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595755    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595758    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595760    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595763    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595766    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595769    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:27:37.595771 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595772    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595775    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595778    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595780    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595783    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595786    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595790    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595795    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595798    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595801    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595804    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595807    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595810    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595813    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595816    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595818    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595821    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595824    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595826    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:27:37.596241 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595829    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595831    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595834    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595837    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595839    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595842    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595845    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595847    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595850    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595853    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595855    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595858    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595860    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595863    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595868    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595871    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595874    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595876    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595879    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595882    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:27:37.596715 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595884    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595887    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595889    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595892    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595895    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595897    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595900    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595902    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595905    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595909    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595911    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595914    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595916    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595919    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595921    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595924    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595926    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595929    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595931    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595934    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:27:37.597211 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595937    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595939    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595942    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595944    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595947    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595949    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595953    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595958    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595962    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595965    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595968    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595970    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595973    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595976    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595980    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595983    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595986    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595989    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.595992    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:27:37.597746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596684    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596690    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596693    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596696    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596701    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596705    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596708    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596711    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596714    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596717    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596719    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596723    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596726    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596729    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596732    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596735    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596738    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596740    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596743    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596746    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:27:37.598217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596749    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596753    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596755    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596758    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596760    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596763    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596766    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596768    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596771    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596773    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596776    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596779    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596781    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596784    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596786    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596789    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596791    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596794    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596796    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596799    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:27:37.598716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596802    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596804    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596807    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596811    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596815    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596819    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596822    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596826    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596828    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596831    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596834    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596837    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596840    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596843    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596845    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596848    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596850    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596853    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596855    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:27:37.599202 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596858    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596860    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596863    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596865    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596868    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596871    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596873    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596876    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596878    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596881    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596884    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596886    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596889    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596892    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596894    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596897    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596900    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596904    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596907    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:27:37.599686 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596909    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596912    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596915    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596917    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596920    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596923    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596926    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596929    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596932    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596934    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596937    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596939    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596942    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596944    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596947    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596949    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596952    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596955    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596957    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596960    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:27:37.600136 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.596963    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597322    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597343    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597348    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597350    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597353    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597356    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597358    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597361    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597363    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597366    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597369    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597371    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597375    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597377    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597380    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597383    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597386    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597388    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597391    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:27:37.600713 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597394    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597397    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597401    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597404    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597407    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597410    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597413    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597416    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597419    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597421    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597424    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597427    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597430    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597432    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597435    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597437    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597440    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597442    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597445    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:27:37.601201 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597448    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597450    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597453    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597456    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597459    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597461    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597463    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597466    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597469    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597472    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597474    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597477    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597479    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597482    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597484    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597487    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597489    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597492    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597494    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597496    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:27:37.601690 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597499    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597501    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597504    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597506    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597509    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597511    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597514    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597518    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597520    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597523    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597525    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597528    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597533    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597535    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597538    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597540    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597543    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597545    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597548    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597550    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:27:37.602185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597554    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597556    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597559    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597562    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597564    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597567    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597570    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597572    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597575    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597578    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597580    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597583    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597586    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597589    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597591    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597594    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597596    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597599    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597602    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:27:37.602720 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597604    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.597609    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597678    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597711    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597718    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597725    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597729    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597732    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597740    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597745    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597748    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597751    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597755    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597758    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597763    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597766    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597768    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597771    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597774    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597777    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597781    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597784    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597787    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:27:37.603165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597791    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597794    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597797    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597800    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597803    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597807    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597810    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597815    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597818    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597821    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597824    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597828    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597831    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597834    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597838    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597841    2580 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597844    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597848    2580 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597851    2580 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597854    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597857    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597860    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597864    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597867    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597870    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:27:37.603742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597873    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597876    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597880    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597883    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597886    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597888    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597891    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597894    2580 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597898    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597901    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597904    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597907    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597910    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597913    2580 flags.go:64] FLAG: --help="false"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597917    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-141-16.ec2.internal"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597920    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597923    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597926    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597929    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597932    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597935    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597939    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597942    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:27:37.604373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597945    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597948    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597951    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597954    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597956    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597959    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597962    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597965    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597968    2580 flags.go:64] FLAG: --lock-file=""
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597971    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597974    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597977    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597982    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597985    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597988    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597991    2580 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597994    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.597997    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598000    2580 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598003    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598008    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598010    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598014    2580 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598018    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598021    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:27:37.604955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598024    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598027    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598030    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598033    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598036    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598043    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598046    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598049    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598053    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598055    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598060    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598063    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598066    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598069    2580 flags.go:64] FLAG: --port="10250"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598072    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598074    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0980f95e3104d2298"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598078    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598081    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598084    2580 flags.go:64] FLAG: --register-node="true"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598087    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598090    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598093    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598096    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598098    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598102    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:27:37.605574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598105    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598108    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598111    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598114    2580 flags.go:64] FLAG: --runonce="false"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598117    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598121    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598124    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598126    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598129    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598132    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598135    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598147    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598152    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598156    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598159    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598162    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598165    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598168    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598171    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598177    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598180    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598183    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598187    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598190    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598192    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:27:37.606312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598196    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598199    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598202    2580 flags.go:64] FLAG: --v="2"
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598206    2580 flags.go:64] FLAG: --version="false"
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598209    2580 flags.go:64] FLAG: --vmodule=""
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598213    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598216    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598340    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598345    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598348    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598350    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598353    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598357    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598360    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598363    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598366    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598368    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598371    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598374    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598377    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598381    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:27:37.607234 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598385    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598389    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598392    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598395    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598397    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598400    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598404    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598408    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598411    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598414    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598417    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598420    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598423    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598426    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598428    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598431    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598434    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598436    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598439    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598441    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:27:37.607782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598444    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598446    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598449    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598456    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598459    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598462    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598465    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598467    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598470    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598472    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598475    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598478    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598481    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598485    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598489    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598492    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598495    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598498    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598501    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:27:37.608271 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598505    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598509    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598514    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598518    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598524    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598530    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598533    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598536    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598539    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598542    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598545    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598547    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598550    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598553    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598555    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598558    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598562    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598564    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598567    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598570    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:27:37.608825 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598572    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598575    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598577    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598580    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598584    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598587    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598589    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598593    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598597    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598602    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598606    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598609    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598611    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598614    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598617    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598620    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598622    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598625    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598628    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598631    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:27:37.609317 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598633    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598636    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598639    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598642    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598644    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.598647    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.598653    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.607926    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.607940    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.607956    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.607962    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608015    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608020    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608023    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608027    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:27:37.609846 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608030    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608033    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608037    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608040    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608043    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608046    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608049    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608051    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608054    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608057    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608060    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608063    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608065    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608068    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608071    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608073    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608076    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608079    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608081    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608084    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:27:37.610227 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608086    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608089    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608091    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608094    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608097    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608100    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608103    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608106    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608108    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608111    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608114    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608116    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608119    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608121    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608124    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608127    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608129    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608132    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608134    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608137    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:27:37.610723 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608139    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608142    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608145    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608147    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608150    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608152    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608155    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608157    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608160    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608162    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608165    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608168    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608170    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608173    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608175    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608178    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608181    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608183    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608186    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608189    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:27:37.611217 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608192    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608194    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608197    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608200    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608203    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608206    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608208    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608211    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608213    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608216    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608219    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608223    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608226    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608229    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608232    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608236    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608239    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608242    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608245    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:27:37.611719 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608248    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608251    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608254    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608257    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608259    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608263    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608267    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608269    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608272    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608275    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608278    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608282    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608285    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608289    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608292    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:27:37.612177 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608295    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.608300    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608440    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608447    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608451    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608454    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608457    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608460    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608463    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608465    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608468    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608470    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608473    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608475    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608479    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:27:37.612609 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608482    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608484    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608487    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608490    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608493    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608497    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608500    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608503    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608506    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608509    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608512    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608515    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608517    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608520    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608522    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608525    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608528    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608531    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608533    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:27:37.612970 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608536    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608539    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608542    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608545    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608547    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608550    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608553    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608555    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608558    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608560    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608562    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608565    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608567    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608571    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608573    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608576    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608579    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608581    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608584    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608586    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:27:37.613452 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608589    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608591    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608594    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608596    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608599    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608601    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608604    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608606    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608609    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608612    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608614    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608617    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608619    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608622    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608625    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608627    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608630    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608632    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608635    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608638    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:27:37.613920 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608640    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608643    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608645    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608648    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608650    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608653    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608656    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608659    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608663    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608666    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608668    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608671    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608674    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608676    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608678    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608681    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608686    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608688    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608691    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:27:37.614423 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608694    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608696    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608699    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608702    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608704    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608707    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608710    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:37.608712    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.608718    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.608817    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.612484    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.613471    2580 server.go:1019] "Starting client certificate rotation"
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.613572    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:27:37.614881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.613612    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:27:37.616085 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.616071    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:27:37.620947 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.620927    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:27:37.633878 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.633860    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:27:37.637669 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.637651    2580 log.go:25] "Validated CRI v1 image API"
Sep 16 18:27:37.638619 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.638601    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:27:37.642742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.642722    2580 fs.go:135] Filesystem UUIDs: map[1a25e13b-1856-4147-98aa-a9f27093c5b7:/dev/nvme0n1p3 32c562f2-b0f9-41dc-a489-82bb8b0f8eb1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 16 18:27:37.642829 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.642742    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:27:37.644634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.644613    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:27:37.647718 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.647607    2580 manager.go:216] Machine: {Timestamp:2026-09-16 18:27:37.647120765 +0000 UTC m=+0.356360148 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099430 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23bfd40226c42f2a1561d4521496f6 SystemUUID:ec23bfd4-0226-c42f-2a15-61d4521496f6 BootID:83744419-043f-4cb4-b8df-4e1d52f71ff7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fa:81:d2:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fa:81:d2:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:9e:3d:31:1b:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:27:37.647718 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.647706    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:27:37.647881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.647809    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:27:37.649510 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.649485    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:27:37.649649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.649518    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-16.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:27:37.650455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.650445    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:27:37.650488 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.650458    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:27:37.650488 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.650465    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:27:37.651353 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.651342    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:27:37.652406 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.652396    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:27:37.655514 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.655503    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:27:37.655637 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.655628    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:27:37.655668 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.655662    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:27:37.655695 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.655679    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:27:37.655695 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.655693    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:27:37.655752 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.655702    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:27:37.655752 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.655709    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:27:37.657149 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.657126    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:27:37.657244 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.657163    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:27:37.659816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.659797    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:27:37.662076 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.662062    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:27:37.662135 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.662091    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:27:37.665293 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665280    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:27:37.665378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665297    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:27:37.665378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665303    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:27:37.665378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665308    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:27:37.665378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665317    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:27:37.665378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665324    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:27:37.665378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665350    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:27:37.665378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665355    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:27:37.665378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665362    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:27:37.665378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665367    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:27:37.665378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665374    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:27:37.665378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.665379    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:27:37.666016 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.666002    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:27:37.666053 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.666022    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:27:37.666320 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.666302    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:27:37.666444 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.666425    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:27:37.668205 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.668191    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-16.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:27:37.669590 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.669579    2580 server.go:1268] "Started kubelet"
Sep 16 18:27:37.669677 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.669653    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:27:37.669735 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.669722    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:27:37.669781 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.669656    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:27:37.670460 ip-10-0-141-16 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:27:37.671924 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.671788    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:27:37.672002 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.671842    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:27:37.674075 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.674058    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:27:37.677604 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.677587    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:27:37.678270 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.678254    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:27:37.678398 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.678376    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-16.ec2.internal\" not found"
Sep 16 18:27:37.678999 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.678976    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:27:37.679073 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.678977    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:27:37.679073 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.679016    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:27:37.679135 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.679127    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:27:37.679166 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.679138    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:27:37.679166 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.679141    2580 factory.go:55] Registering systemd factory
Sep 16 18:27:37.679166 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.679157    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:27:37.679388 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.679373    2580 factory.go:153] Registering CRI-O factory
Sep 16 18:27:37.679426 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.679390    2580 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:27:37.679460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.679440    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:27:37.679492 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.679459    2580 factory.go:103] Registering Raw factory
Sep 16 18:27:37.679492 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.679479    2580 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:27:37.679986 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.679970    2580 manager.go:349] Starting recovery of all containers
Sep 16 18:27:37.683746 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.683703    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:27:37.683866 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.683844    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:27:37.685187 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.683947    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-16.ec2.internal.18d5e07dc1613d94  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-16.ec2.internal,UID:ip-10-0-141-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-16.ec2.internal,},FirstTimestamp:2026-09-16 18:27:37.66954946 +0000 UTC m=+0.378788843,LastTimestamp:2026-09-16 18:27:37.66954946 +0000 UTC m=+0.378788843,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-16.ec2.internal,}"
Sep 16 18:27:37.691316 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.691296    2580 manager.go:354] Recovery completed
Sep 16 18:27:37.696583 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.696569    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:27:37.698408 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.698389    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:27:37.698481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.698426    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:27:37.698481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.698444    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:27:37.698902 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.698891    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:27:37.698902 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.698900    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:27:37.699024 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.698902    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:27:37.699024 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.698915    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:27:37.699556 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.699542    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:27:37.699601 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.699565    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:27:37.699601 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.699578    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:27:37.700065 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.700054    2580 policy_none.go:49] "None policy: Start"
Sep 16 18:27:37.700099 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.700069    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:27:37.700099 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.700079    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:27:37.700483 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.700393    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-16.ec2.internal.18d5e07dc31999fb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-16.ec2.internal,UID:ip-10-0-141-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-16.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-16.ec2.internal,},FirstTimestamp:2026-09-16 18:27:37.698408955 +0000 UTC m=+0.407648342,LastTimestamp:2026-09-16 18:27:37.698408955 +0000 UTC m=+0.407648342,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-16.ec2.internal,}"
Sep 16 18:27:37.700858 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.700845    2580 policy_none.go:47] "Start"
Sep 16 18:27:37.708175 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.708115    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-16.ec2.internal.18d5e07dc319fdcd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-16.ec2.internal,UID:ip-10-0-141-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-16.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-16.ec2.internal,},FirstTimestamp:2026-09-16 18:27:37.698434509 +0000 UTC m=+0.407673896,LastTimestamp:2026-09-16 18:27:37.698434509 +0000 UTC m=+0.407673896,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-16.ec2.internal,}"
Sep 16 18:27:37.715105 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.715081    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kpg6j"
Sep 16 18:27:37.716832 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.716763    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-16.ec2.internal.18d5e07dc31a3bbf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-16.ec2.internal,UID:ip-10-0-141-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-16.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-16.ec2.internal,},FirstTimestamp:2026-09-16 18:27:37.698450367 +0000 UTC m=+0.407689751,LastTimestamp:2026-09-16 18:27:37.698450367 +0000 UTC m=+0.407689751,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-16.ec2.internal,}"
Sep 16 18:27:37.721178 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.721157    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kpg6j"
Sep 16 18:27:37.726951 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.726889    2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-16.ec2.internal.18d5e07dc31999fb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-16.ec2.internal.18d5e07dc31999fb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-16.ec2.internal,UID:ip-10-0-141-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-16.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-16.ec2.internal,},FirstTimestamp:2026-09-16 18:27:37.698408955 +0000 UTC m=+0.407648342,LastTimestamp:2026-09-16 18:27:37.699555051 +0000 UTC m=+0.408794438,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-16.ec2.internal,}"
Sep 16 18:27:37.730952 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.730934    2580 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:27:37.745039 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.730981    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:27:37.745039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.731000    2580 server.go:87] "Starting device plugin registration server"
Sep 16 18:27:37.745039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.731345    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:27:37.745039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.731359    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:27:37.745039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.731710    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:27:37.745039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.731777    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:27:37.745039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.731786    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:27:37.745039 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.732539    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:27:37.745039 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.732577    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-16.ec2.internal\" not found"
Sep 16 18:27:37.775121 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.775098    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:27:37.776382 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.776368    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:27:37.776447 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.776416    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:27:37.776447 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.776437    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:27:37.776530 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.776478    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:27:37.779016 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.778994    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:27:37.832305 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.832276    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:27:37.833239 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.833222    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:27:37.833374 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.833254    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:27:37.833374 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.833269    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:27:37.833374 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.833296    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-16.ec2.internal"
Sep 16 18:27:37.842341 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.842258    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-16.ec2.internal"
Sep 16 18:27:37.842341 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.842279    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-16.ec2.internal\": node \"ip-10-0-141-16.ec2.internal\" not found"
Sep 16 18:27:37.869181 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.869150    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-16.ec2.internal\" not found"
Sep 16 18:27:37.877209 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.877187    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-16.ec2.internal"]
Sep 16 18:27:37.877290 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.877264    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:27:37.880158 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.880145    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:27:37.880229 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.880172    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:27:37.880229 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.880186    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:27:37.881222 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.881210    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:27:37.881365 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.881350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:37.881414 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.881381    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:27:37.881993 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.881967    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:27:37.882060 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.882005    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:27:37.882060 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.882020    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:27:37.882138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.882063    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:27:37.882138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.882088    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:27:37.882138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.882098    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:27:37.883076 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.883063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:37.883125 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.883091    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:27:37.883815 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.883802    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:27:37.883815 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.883824    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:27:37.883905 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.883834    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:27:37.906058 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.906036    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-16.ec2.internal\" not found" node="ip-10-0-141-16.ec2.internal"
Sep 16 18:27:37.910653 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.910637    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-16.ec2.internal\" not found" node="ip-10-0-141-16.ec2.internal"
Sep 16 18:27:37.970257 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:37.970231    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-16.ec2.internal\" not found"
Sep 16 18:27:37.980284 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.980261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1c5b3bf68b93fbd5e401d9ea7792911-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal\" (UID: \"e1c5b3bf68b93fbd5e401d9ea7792911\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:37.980377 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.980289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c5b3bf68b93fbd5e401d9ea7792911-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal\" (UID: \"e1c5b3bf68b93fbd5e401d9ea7792911\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:37.980377 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:37.980305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f3efc08d1fe8fd9d7d17f4d752bb79b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-16.ec2.internal\" (UID: \"6f3efc08d1fe8fd9d7d17f4d752bb79b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:38.070548 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:38.070509    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-16.ec2.internal\" not found"
Sep 16 18:27:38.080877 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.080856    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1c5b3bf68b93fbd5e401d9ea7792911-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal\" (UID: \"e1c5b3bf68b93fbd5e401d9ea7792911\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:38.080935 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.080897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1c5b3bf68b93fbd5e401d9ea7792911-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal\" (UID: \"e1c5b3bf68b93fbd5e401d9ea7792911\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:38.080968 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.080934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c5b3bf68b93fbd5e401d9ea7792911-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal\" (UID: \"e1c5b3bf68b93fbd5e401d9ea7792911\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:38.080968 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.080952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f3efc08d1fe8fd9d7d17f4d752bb79b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-16.ec2.internal\" (UID: \"6f3efc08d1fe8fd9d7d17f4d752bb79b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:38.081028 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.080977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f3efc08d1fe8fd9d7d17f4d752bb79b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-16.ec2.internal\" (UID: \"6f3efc08d1fe8fd9d7d17f4d752bb79b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:38.081028 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.081014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c5b3bf68b93fbd5e401d9ea7792911-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal\" (UID: \"e1c5b3bf68b93fbd5e401d9ea7792911\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:38.171310 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:38.171233    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-16.ec2.internal\" not found"
Sep 16 18:27:38.208670 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.208650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:38.214036 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.214017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:38.271382 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:38.271354    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-16.ec2.internal\" not found"
Sep 16 18:27:38.371888 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:38.371859    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-16.ec2.internal\" not found"
Sep 16 18:27:38.472639 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:38.472570    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-16.ec2.internal\" not found"
Sep 16 18:27:38.576995 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:38.576966    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-16.ec2.internal\" not found"
Sep 16 18:27:38.612998 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.612976    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:27:38.613475 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.613119    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:27:38.667321 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.667292    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:27:38.677807 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.677782    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:27:38.678575 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.678555    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:38.688228 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.688205    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:27:38.697026 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.697008    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:27:38.698985 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.698965    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-16.ec2.internal"
Sep 16 18:27:38.706966 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.706948    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:27:38.716288 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:38.716257    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c5b3bf68b93fbd5e401d9ea7792911.slice/crio-711d1d2f129c5c09b1c4ca51c523f64efcec1ba1bd7d39adca3ec70f278ff3d2 WatchSource:0}: Error finding container 711d1d2f129c5c09b1c4ca51c523f64efcec1ba1bd7d39adca3ec70f278ff3d2: Status 404 returned error can't find the container with id 711d1d2f129c5c09b1c4ca51c523f64efcec1ba1bd7d39adca3ec70f278ff3d2
Sep 16 18:27:38.717294 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:38.717263    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3efc08d1fe8fd9d7d17f4d752bb79b.slice/crio-77b890741ed291d03f4a962b23f85f45136b7ad7369026595ee105ea2fe638e1 WatchSource:0}: Error finding container 77b890741ed291d03f4a962b23f85f45136b7ad7369026595ee105ea2fe638e1: Status 404 returned error can't find the container with id 77b890741ed291d03f4a962b23f85f45136b7ad7369026595ee105ea2fe638e1
Sep 16 18:27:38.717406 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.717387    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gpzsc"
Sep 16 18:27:38.724093 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.724024    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gpzsc"
Sep 16 18:27:38.726380 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.726348    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:22:38 +0000 UTC" deadline="2028-03-18 12:05:12.987996978 +0000 UTC"
Sep 16 18:27:38.726483 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.726381    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13169h37m34.261621453s"
Sep 16 18:27:38.729032 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.729014    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:27:38.767057 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.767029    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:27:38.780799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.780748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal" event={"ID":"e1c5b3bf68b93fbd5e401d9ea7792911","Type":"ContainerStarted","Data":"711d1d2f129c5c09b1c4ca51c523f64efcec1ba1bd7d39adca3ec70f278ff3d2"}
Sep 16 18:27:38.781947 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.781924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-16.ec2.internal" event={"ID":"6f3efc08d1fe8fd9d7d17f4d752bb79b","Type":"ContainerStarted","Data":"77b890741ed291d03f4a962b23f85f45136b7ad7369026595ee105ea2fe638e1"}
Sep 16 18:27:38.866972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:38.866935    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:27:39.522172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.522137    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:27:39.657607 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.657566    2580 apiserver.go:52] "Watching apiserver"
Sep 16 18:27:39.664459 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.664434    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:27:39.665856 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.665821    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal","openshift-multus/multus-fww7f","openshift-network-diagnostics/network-check-target-rxwgv","kube-system/konnectivity-agent-mdgrv","openshift-dns/node-resolver-r5z5r","openshift-multus/multus-additional-cni-plugins-rtj68","openshift-multus/network-metrics-daemon-jd7wp","openshift-network-operator/iptables-alerter-9djtb","openshift-ovn-kubernetes/ovnkube-node-b69vs","kube-system/global-pull-secret-syncer-txjjd","kube-system/kube-apiserver-proxy-ip-10-0-141-16.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d","openshift-cluster-node-tuning-operator/tuned-dtmfm","openshift-image-registry/node-ca-bfn66"]
Sep 16 18:27:39.667375 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.667355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:39.667482 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:39.667442    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:27:39.668744 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.668717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.670415 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.670395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:39.670523 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:39.670463    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:27:39.671681 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.671209    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpt92\""
Sep 16 18:27:39.671681 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.671382    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:27:39.671681 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.671498    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:27:39.671901 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.671793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:27:39.673509 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.673182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r5z5r"
Sep 16 18:27:39.674242 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.674219    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4wtnx\""
Sep 16 18:27:39.674428 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.674397    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:27:39.674671 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.674652    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:27:39.674764 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.674743    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.675729 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.675710    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:27:39.676015 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.675720    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sdm9f\""
Sep 16 18:27:39.676111 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.676018    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:27:39.676756 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.676647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.677817 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.677465    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:27:39.677817 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.677602    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:27:39.677817 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.677708    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:27:39.677983 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.677922    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-svphk\""
Sep 16 18:27:39.678223 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.678048    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:27:39.678223 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.678125    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:27:39.679056 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.679034    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zqg8m\""
Sep 16 18:27:39.679142 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.679117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bfn66"
Sep 16 18:27:39.679625 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.679606    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:27:39.679710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.679642    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:27:39.679710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.679675    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:27:39.679819 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.679608    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:27:39.679819 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.679612    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:27:39.680034 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.680014    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:27:39.680832 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.680812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.681565 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.681508    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:27:39.681565 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.681521    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:27:39.681979 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.681955    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2w6gn\""
Sep 16 18:27:39.682064 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.681966    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:27:39.683051 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.683025    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:39.683617 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.683594    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:27:39.683710 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:39.683638    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:27:39.683767 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.683729    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w6hmp\""
Sep 16 18:27:39.685986 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.685964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9djtb"
Sep 16 18:27:39.687755 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.687732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.688298 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.688270    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:27:39.688910 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.688890    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9hjz7\""
Sep 16 18:27:39.688910 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.688904    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:27:39.689144 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/056dd948-046e-4d8d-bcd6-e96f225b93d5-ovnkube-script-lib\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.689144 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689145    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:27:39.689314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-multus-conf-dir\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.689314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknnx\" (UniqueName: \"kubernetes.io/projected/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-kube-api-access-wknnx\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.689314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.689314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-run-netns\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.689314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-node-log\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.689314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/056dd948-046e-4d8d-bcd6-e96f225b93d5-ovnkube-config\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.689314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-host\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.689690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb761fe6-5e93-438b-a415-2c50316fc733-agent-certs\") pod \"konnectivity-agent-mdgrv\" (UID: \"eb761fe6-5e93-438b-a415-2c50316fc733\") " pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:27:39.689690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-systemd-units\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.689690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/056dd948-046e-4d8d-bcd6-e96f225b93d5-env-overrides\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.689690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-etc-kubernetes\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.689690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-system-cni-dir\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.689690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-var-lib-cni-bin\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.689690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfqps\" (UniqueName: \"kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps\") pod \"network-check-target-rxwgv\" (UID: \"17e3f8a4-0041-47e3-ade2-2744573aaa54\") " pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:39.689690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-cni-netd\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.689690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-var-lib-kubelet\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-modprobe-d\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-sysconfig\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46-hosts-file\") pod \"node-resolver-r5z5r\" (UID: \"7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46\") " pod="openshift-dns/node-resolver-r5z5r"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-var-lib-openvswitch\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2-host\") pod \"node-ca-bfn66\" (UID: \"cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2\") " pod="openshift-image-registry/node-ca-bfn66"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2-serviceca\") pod \"node-ca-bfn66\" (UID: \"cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2\") " pod="openshift-image-registry/node-ca-bfn66"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-multus-socket-dir-parent\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-sys\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.689978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-systemd\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-lib-modules\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm6bt\" (UniqueName: \"kubernetes.io/projected/58e279fd-c891-476e-be69-1802a7262536-kube-api-access-nm6bt\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvkls\" (UniqueName: \"kubernetes.io/projected/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-kube-api-access-tvkls\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.690138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-cnibin\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-run-multus-certs\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-tuned\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-multus-cni-dir\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb761fe6-5e93-438b-a415-2c50316fc733-konnectivity-ca\") pod \"konnectivity-agent-mdgrv\" (UID: \"eb761fe6-5e93-438b-a415-2c50316fc733\") " pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-system-cni-dir\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-cnibin\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-cni-binary-copy\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690318    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-run-systemd\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/056dd948-046e-4d8d-bcd6-e96f225b93d5-ovn-node-metrics-cert\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-hostroot\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690400    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-sysctl-conf\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-tmp\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-kubelet\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.690655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-run-openvswitch\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-run-ovn\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b730f81-def4-434f-903f-cb30aa33272e-cni-binary-copy\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-slash\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690647    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-run-k8s-cni-cncf-io\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-run-netns\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b730f81-def4-434f-903f-cb30aa33272e-multus-daemon-config\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690853    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.690851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pm69\" (UniqueName: \"kubernetes.io/projected/8b730f81-def4-434f-903f-cb30aa33272e-kube-api-access-5pm69\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-kubernetes\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46-tmp-dir\") pod \"node-resolver-r5z5r\" (UID: \"7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46\") " pod="openshift-dns/node-resolver-r5z5r"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx67n\" (UniqueName: \"kubernetes.io/projected/7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46-kube-api-access-jx67n\") pod \"node-resolver-r5z5r\" (UID: \"7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46\") " pod="openshift-dns/node-resolver-r5z5r"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-cni-bin\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-log-socket\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.691392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2xwh\" (UniqueName: \"kubernetes.io/projected/cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2-kube-api-access-g2xwh\") pod \"node-ca-bfn66\" (UID: \"cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2\") " pod="openshift-image-registry/node-ca-bfn66"
Sep 16 18:27:39.692467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-var-lib-kubelet\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.692467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691229    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hwn5q\""
Sep 16 18:27:39.692467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-etc-openvswitch\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.692467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2zjr\" (UniqueName: \"kubernetes.io/projected/056dd948-046e-4d8d-bcd6-e96f225b93d5-kube-api-access-k2zjr\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.692467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-var-lib-cni-multus\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.692467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-sysctl-d\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.692467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.691338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-run\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.726941 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.726888    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:22:39 +0000 UTC" deadline="2028-03-03 07:55:36.507906534 +0000 UTC"
Sep 16 18:27:39.727143 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.727125    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12805h27m56.780789053s"
Sep 16 18:27:39.779944 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.779876    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:27:39.792528 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-cni-netd\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.792528 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-var-lib-kubelet\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.792528 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-modprobe-d\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-sysconfig\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46-hosts-file\") pod \"node-resolver-r5z5r\" (UID: \"7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46\") " pod="openshift-dns/node-resolver-r5z5r"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-var-lib-openvswitch\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2-host\") pod \"node-ca-bfn66\" (UID: \"cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2\") " pod="openshift-image-registry/node-ca-bfn66"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2-serviceca\") pod \"node-ca-bfn66\" (UID: \"cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2\") " pod="openshift-image-registry/node-ca-bfn66"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-multus-socket-dir-parent\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-sys\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-sys-fs\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-systemd\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-lib-modules\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm6bt\" (UniqueName: \"kubernetes.io/projected/58e279fd-c891-476e-be69-1802a7262536-kube-api-access-nm6bt\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkls\" (UniqueName: \"kubernetes.io/projected/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-kube-api-access-tvkls\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-cnibin\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.792799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-run-multus-certs\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-tuned\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-multus-cni-dir\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6479e41-c4ca-420b-8cf0-610ebd939016-host-slash\") pod \"iptables-alerter-9djtb\" (UID: \"c6479e41-c4ca-420b-8cf0-610ebd939016\") " pod="openshift-network-operator/iptables-alerter-9djtb"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb761fe6-5e93-438b-a415-2c50316fc733-konnectivity-ca\") pod \"konnectivity-agent-mdgrv\" (UID: \"eb761fe6-5e93-438b-a415-2c50316fc733\") " pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-system-cni-dir\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-cnibin\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-cni-binary-copy\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-run-systemd\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/056dd948-046e-4d8d-bcd6-e96f225b93d5-ovn-node-metrics-cert\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-hostroot\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.792995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-sysctl-conf\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-tmp\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-registration-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-kubelet\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.793566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-run-openvswitch\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-run-ovn\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b730f81-def4-434f-903f-cb30aa33272e-cni-binary-copy\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-slash\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-run-k8s-cni-cncf-io\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-run-netns\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b730f81-def4-434f-903f-cb30aa33272e-multus-daemon-config\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pm69\" (UniqueName: \"kubernetes.io/projected/8b730f81-def4-434f-903f-cb30aa33272e-kube-api-access-5pm69\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-kubernetes\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-device-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-cni-netd\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be53619d-4600-4740-b307-ab98564ff088-kubelet-config\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46-tmp-dir\") pod \"node-resolver-r5z5r\" (UID: \"7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46\") " pod="openshift-dns/node-resolver-r5z5r"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx67n\" (UniqueName: \"kubernetes.io/projected/7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46-kube-api-access-jx67n\") pod \"node-resolver-r5z5r\" (UID: \"7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46\") " pod="openshift-dns/node-resolver-r5z5r"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-var-lib-kubelet\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.794482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-cni-bin\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-log-socket\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2xwh\" (UniqueName: \"kubernetes.io/projected/cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2-kube-api-access-g2xwh\") pod \"node-ca-bfn66\" (UID: \"cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2\") " pod="openshift-image-registry/node-ca-bfn66"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-var-lib-kubelet\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6479e41-c4ca-420b-8cf0-610ebd939016-iptables-alerter-script\") pod \"iptables-alerter-9djtb\" (UID: \"c6479e41-c4ca-420b-8cf0-610ebd939016\") " pod="openshift-network-operator/iptables-alerter-9djtb"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-etc-openvswitch\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zjr\" (UniqueName: \"kubernetes.io/projected/056dd948-046e-4d8d-bcd6-e96f225b93d5-kube-api-access-k2zjr\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-cnibin\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-var-lib-cni-multus\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-run-multus-certs\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-sysctl-d\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-run\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-socket-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/056dd948-046e-4d8d-bcd6-e96f225b93d5-ovnkube-script-lib\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-multus-conf-dir\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.795329 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wknnx\" (UniqueName: \"kubernetes.io/projected/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-kube-api-access-wknnx\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.793992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjpjf\" (UniqueName: \"kubernetes.io/projected/c6479e41-c4ca-420b-8cf0-610ebd939016-kube-api-access-bjpjf\") pod \"iptables-alerter-9djtb\" (UID: \"c6479e41-c4ca-420b-8cf0-610ebd939016\") " pod="openshift-network-operator/iptables-alerter-9djtb"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-run-netns\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-sysconfig\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-multus-cni-dir\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-run-netns\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46-hosts-file\") pod \"node-resolver-r5z5r\" (UID: \"7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46\") " pod="openshift-dns/node-resolver-r5z5r"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-system-cni-dir\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794455    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb761fe6-5e93-438b-a415-2c50316fc733-konnectivity-ca\") pod \"konnectivity-agent-mdgrv\" (UID: \"eb761fe6-5e93-438b-a415-2c50316fc733\") " pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2-host\") pod \"node-ca-bfn66\" (UID: \"cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2\") " pod="openshift-image-registry/node-ca-bfn66"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-var-lib-openvswitch\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.795037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-cnibin\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.795379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2-serviceca\") pod \"node-ca-bfn66\" (UID: \"cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2\") " pod="openshift-image-registry/node-ca-bfn66"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.795458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-cni-binary-copy\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.795478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-multus-socket-dir-parent\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.796245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.795514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-run-systemd\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.795538    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-sys\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.795602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-systemd\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.795711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-lib-modules\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.795751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-cni-bin\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.795792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-log-socket\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-var-lib-kubelet\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-etc-openvswitch\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-hostroot\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-sysctl-conf\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-var-lib-cni-multus\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:39.796443    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-sysctl-d\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:39.796506    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs podName:58e279fd-c891-476e-be69-1802a7262536 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:40.296488213 +0000 UTC m=+3.005727603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs") pod "network-metrics-daemon-jd7wp" (UID: "58e279fd-c891-476e-be69-1802a7262536") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-kubelet\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-run-openvswitch\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-multus-conf-dir\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.797155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-run-ovn\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.798311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.798311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-run\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.798311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-run-k8s-cni-cncf-io\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.798311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.796926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-run-netns\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.798311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.797390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b730f81-def4-434f-903f-cb30aa33272e-cni-binary-copy\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.798311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.797439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b730f81-def4-434f-903f-cb30aa33272e-multus-daemon-config\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.798311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.797579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-host-slash\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.798311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.794891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.798311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.797670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-node-log\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.800408 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.800377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/056dd948-046e-4d8d-bcd6-e96f225b93d5-ovn-node-metrics-cert\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.800511 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.800440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-modprobe-d\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.800762 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.800740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-tuned\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.801180 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46-tmp-dir\") pod \"node-resolver-r5z5r\" (UID: \"7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46\") " pod="openshift-dns/node-resolver-r5z5r"
Sep 16 18:27:39.801274 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.801274 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/056dd948-046e-4d8d-bcd6-e96f225b93d5-ovnkube-config\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.801274 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-host\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.801510 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-etc-kubernetes\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.801510 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb761fe6-5e93-438b-a415-2c50316fc733-agent-certs\") pod \"konnectivity-agent-mdgrv\" (UID: \"eb761fe6-5e93-438b-a415-2c50316fc733\") " pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:27:39.801510 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-systemd-units\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.801510 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/056dd948-046e-4d8d-bcd6-e96f225b93d5-env-overrides\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.801510 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-etc-kubernetes\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.801790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.801790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmv2\" (UniqueName: \"kubernetes.io/projected/15e524c0-e4ed-4337-8f70-06b943f1bf8b-kube-api-access-plmv2\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.801790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be53619d-4600-4740-b307-ab98564ff088-dbus\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:39.801790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-system-cni-dir\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.801790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-var-lib-cni-bin\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.801790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.801729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqps\" (UniqueName: \"kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps\") pod \"network-check-target-rxwgv\" (UID: \"17e3f8a4-0041-47e3-ade2-2744573aaa54\") " pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:39.802105 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.802084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-etc-kubernetes\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.802160 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.802138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-host\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.802225 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.802187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-node-log\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.803694 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.802699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/056dd948-046e-4d8d-bcd6-e96f225b93d5-ovnkube-config\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.803694 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.802800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-system-cni-dir\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.803694 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.802855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b730f81-def4-434f-903f-cb30aa33272e-host-var-lib-cni-bin\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.803694 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.803015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/056dd948-046e-4d8d-bcd6-e96f225b93d5-env-overrides\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.803694 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.803050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-tmp\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.803694 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.803066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/056dd948-046e-4d8d-bcd6-e96f225b93d5-systemd-units\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.803694 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.803380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/056dd948-046e-4d8d-bcd6-e96f225b93d5-ovnkube-script-lib\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.806295 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.806262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvkls\" (UniqueName: \"kubernetes.io/projected/9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef-kube-api-access-tvkls\") pod \"multus-additional-cni-plugins-rtj68\" (UID: \"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef\") " pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:39.808533 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:39.808511    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:27:39.808618 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:39.808538    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:27:39.808618 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:39.808553    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rfqps for pod openshift-network-diagnostics/network-check-target-rxwgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:39.808738 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:39.808628    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps podName:17e3f8a4-0041-47e3-ade2-2744573aaa54 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:40.30860815 +0000 UTC m=+3.017847536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rfqps" (UniqueName: "kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps") pod "network-check-target-rxwgv" (UID: "17e3f8a4-0041-47e3-ade2-2744573aaa54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:39.810475 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.810386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknnx\" (UniqueName: \"kubernetes.io/projected/9b6be9c0-baec-4e10-95e0-5e4a91b3acda-kube-api-access-wknnx\") pod \"tuned-dtmfm\" (UID: \"9b6be9c0-baec-4e10-95e0-5e4a91b3acda\") " pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.812238 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.811769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb761fe6-5e93-438b-a415-2c50316fc733-agent-certs\") pod \"konnectivity-agent-mdgrv\" (UID: \"eb761fe6-5e93-438b-a415-2c50316fc733\") " pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:27:39.812238 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.811807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pm69\" (UniqueName: \"kubernetes.io/projected/8b730f81-def4-434f-903f-cb30aa33272e-kube-api-access-5pm69\") pod \"multus-fww7f\" (UID: \"8b730f81-def4-434f-903f-cb30aa33272e\") " pod="openshift-multus/multus-fww7f"
Sep 16 18:27:39.812238 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.812181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm6bt\" (UniqueName: \"kubernetes.io/projected/58e279fd-c891-476e-be69-1802a7262536-kube-api-access-nm6bt\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:39.812238 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.812206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2xwh\" (UniqueName: \"kubernetes.io/projected/cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2-kube-api-access-g2xwh\") pod \"node-ca-bfn66\" (UID: \"cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2\") " pod="openshift-image-registry/node-ca-bfn66"
Sep 16 18:27:39.812430 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.812289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2zjr\" (UniqueName: \"kubernetes.io/projected/056dd948-046e-4d8d-bcd6-e96f225b93d5-kube-api-access-k2zjr\") pod \"ovnkube-node-b69vs\" (UID: \"056dd948-046e-4d8d-bcd6-e96f225b93d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:39.813454 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.813429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx67n\" (UniqueName: \"kubernetes.io/projected/7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46-kube-api-access-jx67n\") pod \"node-resolver-r5z5r\" (UID: \"7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46\") " pod="openshift-dns/node-resolver-r5z5r"
Sep 16 18:27:39.902917 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.902880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjpjf\" (UniqueName: \"kubernetes.io/projected/c6479e41-c4ca-420b-8cf0-610ebd939016-kube-api-access-bjpjf\") pod \"iptables-alerter-9djtb\" (UID: \"c6479e41-c4ca-420b-8cf0-610ebd939016\") " pod="openshift-network-operator/iptables-alerter-9djtb"
Sep 16 18:27:39.902917 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.902920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903134 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.902950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903134 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.902995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmv2\" (UniqueName: \"kubernetes.io/projected/15e524c0-e4ed-4337-8f70-06b943f1bf8b-kube-api-access-plmv2\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903134 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903134 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be53619d-4600-4740-b307-ab98564ff088-dbus\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:39.903134 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-sys-fs\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903134 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6479e41-c4ca-420b-8cf0-610ebd939016-host-slash\") pod \"iptables-alerter-9djtb\" (UID: \"c6479e41-c4ca-420b-8cf0-610ebd939016\") " pod="openshift-network-operator/iptables-alerter-9djtb"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-registration-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be53619d-4600-4740-b307-ab98564ff088-dbus\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6479e41-c4ca-420b-8cf0-610ebd939016-host-slash\") pod \"iptables-alerter-9djtb\" (UID: \"c6479e41-c4ca-420b-8cf0-610ebd939016\") " pod="openshift-network-operator/iptables-alerter-9djtb"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-registration-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-device-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be53619d-4600-4740-b307-ab98564ff088-kubelet-config\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6479e41-c4ca-420b-8cf0-610ebd939016-iptables-alerter-script\") pod \"iptables-alerter-9djtb\" (UID: \"c6479e41-c4ca-420b-8cf0-610ebd939016\") " pod="openshift-network-operator/iptables-alerter-9djtb"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-socket-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-sys-fs\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-device-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903460 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be53619d-4600-4740-b307-ab98564ff088-kubelet-config\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:39.903866 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/15e524c0-e4ed-4337-8f70-06b943f1bf8b-socket-dir\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.903866 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:39.903504    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:39.903866 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:39.903559    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret podName:be53619d-4600-4740-b307-ab98564ff088 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:40.40354619 +0000 UTC m=+3.112785566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret") pod "global-pull-secret-syncer-txjjd" (UID: "be53619d-4600-4740-b307-ab98564ff088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:39.903866 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.903808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6479e41-c4ca-420b-8cf0-610ebd939016-iptables-alerter-script\") pod \"iptables-alerter-9djtb\" (UID: \"c6479e41-c4ca-420b-8cf0-610ebd939016\") " pod="openshift-network-operator/iptables-alerter-9djtb"
Sep 16 18:27:39.913989 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.913967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjpjf\" (UniqueName: \"kubernetes.io/projected/c6479e41-c4ca-420b-8cf0-610ebd939016-kube-api-access-bjpjf\") pod \"iptables-alerter-9djtb\" (UID: \"c6479e41-c4ca-420b-8cf0-610ebd939016\") " pod="openshift-network-operator/iptables-alerter-9djtb"
Sep 16 18:27:39.914117 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.914045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmv2\" (UniqueName: \"kubernetes.io/projected/15e524c0-e4ed-4337-8f70-06b943f1bf8b-kube-api-access-plmv2\") pod \"aws-ebs-csi-driver-node-bbz6d\" (UID: \"15e524c0-e4ed-4337-8f70-06b943f1bf8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:39.983401 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.983362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dtmfm"
Sep 16 18:27:39.995539 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:39.995511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r5z5r"
Sep 16 18:27:40.003548 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.003519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:27:40.011371 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.011348    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rtj68"
Sep 16 18:27:40.018285 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.018268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:27:40.024897 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.024876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bfn66"
Sep 16 18:27:40.033483 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.033437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fww7f"
Sep 16 18:27:40.041875 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.041856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9djtb"
Sep 16 18:27:40.047549 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.047530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d"
Sep 16 18:27:40.306011 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.305920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:40.306171 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:40.306091    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:40.306171 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:40.306164    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs podName:58e279fd-c891-476e-be69-1802a7262536 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:41.306146771 +0000 UTC m=+4.015386165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs") pod "network-metrics-daemon-jd7wp" (UID: "58e279fd-c891-476e-be69-1802a7262536") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:40.406863 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.406823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqps\" (UniqueName: \"kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps\") pod \"network-check-target-rxwgv\" (UID: \"17e3f8a4-0041-47e3-ade2-2744573aaa54\") " pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:40.407064 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.406920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:40.407064 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:40.407028    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:40.407064 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:40.407033    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:27:40.407064 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:40.407063    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:27:40.407281 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:40.407075    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rfqps for pod openshift-network-diagnostics/network-check-target-rxwgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:40.407281 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:40.407095    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret podName:be53619d-4600-4740-b307-ab98564ff088 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:41.407076873 +0000 UTC m=+4.116316245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret") pod "global-pull-secret-syncer-txjjd" (UID: "be53619d-4600-4740-b307-ab98564ff088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:40.407281 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:40.407127    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps podName:17e3f8a4-0041-47e3-ade2-2744573aaa54 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:41.407108944 +0000 UTC m=+4.116348320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfqps" (UniqueName: "kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps") pod "network-check-target-rxwgv" (UID: "17e3f8a4-0041-47e3-ade2-2744573aaa54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:40.728020 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.727909    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:22:39 +0000 UTC" deadline="2028-06-15 11:28:48.384195736 +0000 UTC"
Sep 16 18:27:40.728020 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.727966    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15305h1m7.656234523s"
Sep 16 18:27:40.777549 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.777510    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:40.777549 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:40.777533    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:40.777780 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:40.777653    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:27:40.777841 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:40.777819    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:27:40.859661 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:40.859612    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056dd948_046e_4d8d_bcd6_e96f225b93d5.slice/crio-afd53837abc14e58e7243cbc51d9b785b7b91c4a1143b754e6d7873c3b308390 WatchSource:0}: Error finding container afd53837abc14e58e7243cbc51d9b785b7b91c4a1143b754e6d7873c3b308390: Status 404 returned error can't find the container with id afd53837abc14e58e7243cbc51d9b785b7b91c4a1143b754e6d7873c3b308390
Sep 16 18:27:40.870610 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:40.870583    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8b1fa8_e5b5_4101_b7ad_f46531ed7eef.slice/crio-6a54b078282a7602e35e4360687247e6b4d14228052414f6cd27862db5256fbb WatchSource:0}: Error finding container 6a54b078282a7602e35e4360687247e6b4d14228052414f6cd27862db5256fbb: Status 404 returned error can't find the container with id 6a54b078282a7602e35e4360687247e6b4d14228052414f6cd27862db5256fbb
Sep 16 18:27:40.873078 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:40.873054    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb761fe6_5e93_438b_a415_2c50316fc733.slice/crio-ea40c9f27d9c32df307a546e9581d0851307000b3a884b8b8e219347621e271d WatchSource:0}: Error finding container ea40c9f27d9c32df307a546e9581d0851307000b3a884b8b8e219347621e271d: Status 404 returned error can't find the container with id ea40c9f27d9c32df307a546e9581d0851307000b3a884b8b8e219347621e271d
Sep 16 18:27:40.875131 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:40.875005    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6479e41_c4ca_420b_8cf0_610ebd939016.slice/crio-85904b97f69d23293f285d922eca214788d4ec8446be91861c8b6c2456f786a1 WatchSource:0}: Error finding container 85904b97f69d23293f285d922eca214788d4ec8446be91861c8b6c2456f786a1: Status 404 returned error can't find the container with id 85904b97f69d23293f285d922eca214788d4ec8446be91861c8b6c2456f786a1
Sep 16 18:27:40.877263 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:40.877243    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e524c0_e4ed_4337_8f70_06b943f1bf8b.slice/crio-b8ae0c6d846d3a54db4faaa21f3e45395c236dade7dee03ea24725110f383e1f WatchSource:0}: Error finding container b8ae0c6d846d3a54db4faaa21f3e45395c236dade7dee03ea24725110f383e1f: Status 404 returned error can't find the container with id b8ae0c6d846d3a54db4faaa21f3e45395c236dade7dee03ea24725110f383e1f
Sep 16 18:27:40.878735 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:27:40.878684    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdbe8e9_3ed9_49e4_a10d_41c1699e7a46.slice/crio-f73065454381b7315578750e01a06a536be8527353017f9cc8616a8c4f38f21c WatchSource:0}: Error finding container f73065454381b7315578750e01a06a536be8527353017f9cc8616a8c4f38f21c: Status 404 returned error can't find the container with id f73065454381b7315578750e01a06a536be8527353017f9cc8616a8c4f38f21c
Sep 16 18:27:41.314737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.314701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:41.314900 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:41.314870    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:41.314950 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:41.314941    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs podName:58e279fd-c891-476e-be69-1802a7262536 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:43.314924186 +0000 UTC m=+6.024163563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs") pod "network-metrics-daemon-jd7wp" (UID: "58e279fd-c891-476e-be69-1802a7262536") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:41.415585 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.415497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:41.415585 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.415564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqps\" (UniqueName: \"kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps\") pod \"network-check-target-rxwgv\" (UID: \"17e3f8a4-0041-47e3-ade2-2744573aaa54\") " pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:41.415788 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:41.415713    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:27:41.415788 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:41.415731    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:27:41.415788 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:41.415745    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rfqps for pod openshift-network-diagnostics/network-check-target-rxwgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:41.415936 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:41.415809    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps podName:17e3f8a4-0041-47e3-ade2-2744573aaa54 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:43.415788964 +0000 UTC m=+6.125028342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfqps" (UniqueName: "kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps") pod "network-check-target-rxwgv" (UID: "17e3f8a4-0041-47e3-ade2-2744573aaa54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:41.416362 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:41.416246    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:41.416362 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:41.416317    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret podName:be53619d-4600-4740-b307-ab98564ff088 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:43.416297373 +0000 UTC m=+6.125536766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret") pod "global-pull-secret-syncer-txjjd" (UID: "be53619d-4600-4740-b307-ab98564ff088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:41.780695 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.780169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:41.780695 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:41.780303    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:27:41.805101 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.804956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r5z5r" event={"ID":"7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46","Type":"ContainerStarted","Data":"f73065454381b7315578750e01a06a536be8527353017f9cc8616a8c4f38f21c"}
Sep 16 18:27:41.821737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.821657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mdgrv" event={"ID":"eb761fe6-5e93-438b-a415-2c50316fc733","Type":"ContainerStarted","Data":"ea40c9f27d9c32df307a546e9581d0851307000b3a884b8b8e219347621e271d"}
Sep 16 18:27:41.832453 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.832358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtj68" event={"ID":"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef","Type":"ContainerStarted","Data":"6a54b078282a7602e35e4360687247e6b4d14228052414f6cd27862db5256fbb"}
Sep 16 18:27:41.846438 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.846354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-16.ec2.internal" event={"ID":"6f3efc08d1fe8fd9d7d17f4d752bb79b","Type":"ContainerStarted","Data":"6e38da06fdaa837ff19c593a78497fffcaabd995aaaf5160e97dcde8c350e121"}
Sep 16 18:27:41.848654 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.848625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dtmfm" event={"ID":"9b6be9c0-baec-4e10-95e0-5e4a91b3acda","Type":"ContainerStarted","Data":"8a54dd797950b938ba84f883994df0fd9988e707ec4fed31e8bf2e81a2f53bdc"}
Sep 16 18:27:41.852172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.852145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9djtb" event={"ID":"c6479e41-c4ca-420b-8cf0-610ebd939016","Type":"ContainerStarted","Data":"85904b97f69d23293f285d922eca214788d4ec8446be91861c8b6c2456f786a1"}
Sep 16 18:27:41.865118 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.865091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d" event={"ID":"15e524c0-e4ed-4337-8f70-06b943f1bf8b","Type":"ContainerStarted","Data":"b8ae0c6d846d3a54db4faaa21f3e45395c236dade7dee03ea24725110f383e1f"}
Sep 16 18:27:41.879991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.879900    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs" event={"ID":"056dd948-046e-4d8d-bcd6-e96f225b93d5","Type":"ContainerStarted","Data":"afd53837abc14e58e7243cbc51d9b785b7b91c4a1143b754e6d7873c3b308390"}
Sep 16 18:27:41.888308 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.888282    2580 generic.go:356] "Generic (PLEG): container finished" podID="e1c5b3bf68b93fbd5e401d9ea7792911" containerID="90eb0b6630d82e91952c6459057089947de63f4d4b6d8381bdd3dc7458dba722" exitCode=0
Sep 16 18:27:41.888462 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.888393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal" event={"ID":"e1c5b3bf68b93fbd5e401d9ea7792911","Type":"ContainerDied","Data":"90eb0b6630d82e91952c6459057089947de63f4d4b6d8381bdd3dc7458dba722"}
Sep 16 18:27:41.893309 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.893244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bfn66" event={"ID":"cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2","Type":"ContainerStarted","Data":"091b483445490cec4f41cb98bc04821d902679dd891b3867d04a35542b633fc9"}
Sep 16 18:27:41.899988 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.899931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fww7f" event={"ID":"8b730f81-def4-434f-903f-cb30aa33272e","Type":"ContainerStarted","Data":"549485bd332f32831d92528f070374a6398b6a80e4a4c6a432b8c3363583ad7d"}
Sep 16 18:27:41.908318 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:41.907188    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-16.ec2.internal" podStartSLOduration=2.907168141 podStartE2EDuration="2.907168141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:27:41.867435329 +0000 UTC m=+4.576674726" watchObservedRunningTime="2026-09-16 18:27:41.907168141 +0000 UTC m=+4.616407542"
Sep 16 18:27:42.778435 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:42.777612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:42.778435 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:42.777759    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:27:42.778435 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:42.778253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:42.778435 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:42.778367    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:27:42.923251 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:42.923122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal" event={"ID":"e1c5b3bf68b93fbd5e401d9ea7792911","Type":"ContainerStarted","Data":"ca1a2e80e72b0e557a85b2efb9e1cf6b72530b15208c4f93c08c6a3c97d99d40"}
Sep 16 18:27:43.338169 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:43.337567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:43.338169 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:43.337734    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:43.338169 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:43.337799    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs podName:58e279fd-c891-476e-be69-1802a7262536 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:47.337781466 +0000 UTC m=+10.047020844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs") pod "network-metrics-daemon-jd7wp" (UID: "58e279fd-c891-476e-be69-1802a7262536") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:43.439222 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:43.438479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:43.439222 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:43.438536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqps\" (UniqueName: \"kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps\") pod \"network-check-target-rxwgv\" (UID: \"17e3f8a4-0041-47e3-ade2-2744573aaa54\") " pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:43.439222 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:43.438686    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:27:43.439222 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:43.438707    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:27:43.439222 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:43.438720    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rfqps for pod openshift-network-diagnostics/network-check-target-rxwgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:43.439222 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:43.438777    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps podName:17e3f8a4-0041-47e3-ade2-2744573aaa54 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:47.438760449 +0000 UTC m=+10.147999841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfqps" (UniqueName: "kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps") pod "network-check-target-rxwgv" (UID: "17e3f8a4-0041-47e3-ade2-2744573aaa54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:43.439222 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:43.438854    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:43.439222 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:43.438888    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret podName:be53619d-4600-4740-b307-ab98564ff088 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:47.438878052 +0000 UTC m=+10.148117436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret") pod "global-pull-secret-syncer-txjjd" (UID: "be53619d-4600-4740-b307-ab98564ff088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:43.777927 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:43.777426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:43.777927 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:43.777551    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:27:44.776860 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:44.776824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:44.777347 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:44.776972    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:27:44.777347 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:44.777130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:44.777347 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:44.777240    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:27:45.778467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:45.778437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:45.778951 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:45.778563    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:27:46.777919 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:46.777128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:46.777919 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:46.777542    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:27:46.777919 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:46.777374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:46.777919 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:46.777741    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:27:47.372647 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:47.372604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:47.373103 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:47.372739    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:47.373103 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:47.372806    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs podName:58e279fd-c891-476e-be69-1802a7262536 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:55.372784285 +0000 UTC m=+18.082023674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs") pod "network-metrics-daemon-jd7wp" (UID: "58e279fd-c891-476e-be69-1802a7262536") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:47.473083 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:47.473050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:47.473083 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:47.473096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqps\" (UniqueName: \"kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps\") pod \"network-check-target-rxwgv\" (UID: \"17e3f8a4-0041-47e3-ade2-2744573aaa54\") " pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:47.473428 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:47.473248    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:47.473428 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:47.473300    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:27:47.473428 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:47.473319    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:27:47.473428 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:47.473340    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret podName:be53619d-4600-4740-b307-ab98564ff088 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:55.473306208 +0000 UTC m=+18.182545599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret") pod "global-pull-secret-syncer-txjjd" (UID: "be53619d-4600-4740-b307-ab98564ff088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:47.473428 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:47.473348    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rfqps for pod openshift-network-diagnostics/network-check-target-rxwgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:47.473428 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:47.473389    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps podName:17e3f8a4-0041-47e3-ade2-2744573aaa54 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:55.473376709 +0000 UTC m=+18.182616086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfqps" (UniqueName: "kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps") pod "network-check-target-rxwgv" (UID: "17e3f8a4-0041-47e3-ade2-2744573aaa54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:47.777508 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:47.777473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:47.777687 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:47.777576    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:27:48.777628 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:48.777588    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:48.777628 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:48.777628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:48.778132 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:48.777730    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:27:48.778132 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:48.777857    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:27:49.778109 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:49.777607    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:49.778109 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:49.777746    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:27:50.777532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:50.777498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:50.777699 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:50.777539    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:50.777699 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:50.777648    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:27:50.777823 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:50.777768    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:27:51.777032 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:51.777000    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:51.777511 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:51.777132    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:27:52.777454 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:52.777414    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:52.777454 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:52.777437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:52.777872 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:52.777574    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:27:52.777872 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:52.777635    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:27:53.777033 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:53.776979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:53.777216 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:53.777143    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:27:54.777587 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:54.777552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:54.777962 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:54.777559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:54.777962 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:54.777652    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:27:54.777962 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:54.777740    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:27:55.433018 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:55.432985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:55.433211 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:55.433145    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:55.433278 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:55.433218    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs podName:58e279fd-c891-476e-be69-1802a7262536 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:11.433198861 +0000 UTC m=+34.142438232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs") pod "network-metrics-daemon-jd7wp" (UID: "58e279fd-c891-476e-be69-1802a7262536") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:27:55.533898 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:55.533860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:55.533898 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:55.533905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqps\" (UniqueName: \"kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps\") pod \"network-check-target-rxwgv\" (UID: \"17e3f8a4-0041-47e3-ade2-2744573aaa54\") " pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:55.534094 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:55.534014    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:55.534094 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:55.534029    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:27:55.534094 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:55.534045    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:27:55.534094 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:55.534057    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rfqps for pod openshift-network-diagnostics/network-check-target-rxwgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:55.534094 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:55.534081    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret podName:be53619d-4600-4740-b307-ab98564ff088 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:11.534062757 +0000 UTC m=+34.243302153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret") pod "global-pull-secret-syncer-txjjd" (UID: "be53619d-4600-4740-b307-ab98564ff088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:27:55.534317 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:55.534097    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps podName:17e3f8a4-0041-47e3-ade2-2744573aaa54 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:11.534088426 +0000 UTC m=+34.243327809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfqps" (UniqueName: "kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps") pod "network-check-target-rxwgv" (UID: "17e3f8a4-0041-47e3-ade2-2744573aaa54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:27:55.777207 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:55.777177    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:55.777388 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:55.777301    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:27:56.777322 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:56.777286    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:56.777769 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:56.777286    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:56.777769 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:56.777444    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:27:56.777769 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:56.777491    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:27:57.778515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:57.778410    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:57.778952 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:57.778533    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:27:58.777114 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.776913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:27:58.777249 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.776970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:27:58.777249 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:58.777215    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:27:58.777379 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:58.777361    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:27:58.962042 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.962014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bfn66" event={"ID":"cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2","Type":"ContainerStarted","Data":"9dd2067a09efa17d1fef7d2882acf5f3e9f94b9c1f6ad25c81f0687d6028056c"}
Sep 16 18:27:58.963891 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.963870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fww7f" event={"ID":"8b730f81-def4-434f-903f-cb30aa33272e","Type":"ContainerStarted","Data":"fdd5100bf0a5c059f53080b89286a73aa0d5c8b27d4f9f0f551676c3f3a0654f"}
Sep 16 18:27:58.965223 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.965202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r5z5r" event={"ID":"7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46","Type":"ContainerStarted","Data":"1fc98757cc219a9072be697cf6c8efeab6e95d901a4b0c9df7fbe5bb39238fe8"}
Sep 16 18:27:58.966537 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.966519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mdgrv" event={"ID":"eb761fe6-5e93-438b-a415-2c50316fc733","Type":"ContainerStarted","Data":"625e562a9806da50105cdb054a48fc0b7118c7e8993708985b737c60519a59be"}
Sep 16 18:27:58.970872 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.970847    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef" containerID="8247b3064be9bb358baab9c6bb28fb40d87a87a0d83da81772e2a00ee7547c72" exitCode=0
Sep 16 18:27:58.970954 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.970925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtj68" event={"ID":"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef","Type":"ContainerDied","Data":"8247b3064be9bb358baab9c6bb28fb40d87a87a0d83da81772e2a00ee7547c72"}
Sep 16 18:27:58.972596 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.972572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dtmfm" event={"ID":"9b6be9c0-baec-4e10-95e0-5e4a91b3acda","Type":"ContainerStarted","Data":"2c43d8bd8f4eca4cd0116893361bdaf1ac89548bc0097748b02b3b31cb00301f"}
Sep 16 18:27:58.976108 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.976086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d" event={"ID":"15e524c0-e4ed-4337-8f70-06b943f1bf8b","Type":"ContainerStarted","Data":"7d9f88f1c357a0163f7a43ee7e0f84e07137d2f6e0eb9b757af8baefe29913e7"}
Sep 16 18:27:58.979430 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.979411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs" event={"ID":"056dd948-046e-4d8d-bcd6-e96f225b93d5","Type":"ContainerStarted","Data":"34b48843f9c10e9ed1eb01682ffbb14ab47d71f43a6308bce5c1b282e2d94caf"}
Sep 16 18:27:58.979524 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.979479    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs" event={"ID":"056dd948-046e-4d8d-bcd6-e96f225b93d5","Type":"ContainerStarted","Data":"483d5d3e228b6ecddc434ac19ae851ae1cd41b0c5921d944f1cf3a279be42873"}
Sep 16 18:27:58.979596 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.979531    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs" event={"ID":"056dd948-046e-4d8d-bcd6-e96f225b93d5","Type":"ContainerStarted","Data":"baa28914805319e22ff64fd3fb4261b847b54dbde63258f79b3e9182aabe5768"}
Sep 16 18:27:58.979596 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.979580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs" event={"ID":"056dd948-046e-4d8d-bcd6-e96f225b93d5","Type":"ContainerStarted","Data":"209832a15267ed9487115183a605e020929d96ed92c6b629ee3d2f6a44868adc"}
Sep 16 18:27:58.982891 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.982855    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bfn66" podStartSLOduration=3.440125209 podStartE2EDuration="20.982843768s" totalImagesPullingTime="17.542718559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:27:58.98226672 +0000 UTC m=+21.691506111" watchObservedRunningTime="2026-09-16 18:27:58.982843768 +0000 UTC m=+21.692083160"
Sep 16 18:27:58.982989 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.982921    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-16.ec2.internal" podStartSLOduration=19.982916307 podStartE2EDuration="19.982916307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:27:42.949918677 +0000 UTC m=+5.659158065" watchObservedRunningTime="2026-09-16 18:27:58.982916307 +0000 UTC m=+21.692155698"
Sep 16 18:27:58.997944 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:58.997911    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mdgrv" podStartSLOduration=7.270964915 podStartE2EDuration="20.997898453s" totalImagesPullingTime="13.726933538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:27:58.997683682 +0000 UTC m=+21.706923085" watchObservedRunningTime="2026-09-16 18:27:58.997898453 +0000 UTC m=+21.707137845"
Sep 16 18:27:59.048261 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:59.048220    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dtmfm" podStartSLOduration=3.487460761 podStartE2EDuration="21.04820471s" totalImagesPullingTime="17.560743949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:27:59.047839233 +0000 UTC m=+21.757078636" watchObservedRunningTime="2026-09-16 18:27:59.04820471 +0000 UTC m=+21.757444100"
Sep 16 18:27:59.048406 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:59.048386    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fww7f" podStartSLOduration=3.453151149 podStartE2EDuration="21.048380943s" totalImagesPullingTime="17.595229794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:27:59.022912744 +0000 UTC m=+21.732152136" watchObservedRunningTime="2026-09-16 18:27:59.048380943 +0000 UTC m=+21.757620335"
Sep 16 18:27:59.064293 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:59.064256    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r5z5r" podStartSLOduration=3.5264123 podStartE2EDuration="21.064243325s" totalImagesPullingTime="17.537831025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:27:59.063873574 +0000 UTC m=+21.773112968" watchObservedRunningTime="2026-09-16 18:27:59.064243325 +0000 UTC m=+21.773482717"
Sep 16 18:27:59.777477 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:59.777445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:27:59.777703 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:27:59.777580    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:27:59.983760 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:59.983724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9djtb" event={"ID":"c6479e41-c4ca-420b-8cf0-610ebd939016","Type":"ContainerStarted","Data":"01c281a58fc92751b8bf1e90bd34c80dde9016fb923c22ade166feb761b3d854"}
Sep 16 18:27:59.987382 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:59.987353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs" event={"ID":"056dd948-046e-4d8d-bcd6-e96f225b93d5","Type":"ContainerStarted","Data":"9b89a6eb1bc26f67dd05f92034ac20cdd8facc011e0e6d15ea6983bfde682d59"}
Sep 16 18:27:59.987504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:27:59.987418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs" event={"ID":"056dd948-046e-4d8d-bcd6-e96f225b93d5","Type":"ContainerStarted","Data":"c90a39fe5fdb9437508600583cef5d01db25a8c7d1b0e46582912e0326d0a1fe"}
Sep 16 18:28:00.005770 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:00.005712    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9djtb" podStartSLOduration=4.463698906 podStartE2EDuration="22.005687074s" totalImagesPullingTime="17.541988168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:00.005584605 +0000 UTC m=+22.714823998" watchObservedRunningTime="2026-09-16 18:28:00.005687074 +0000 UTC m=+22.714926467"
Sep 16 18:28:00.124646 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:00.124464    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:28:00.744656 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:00.744518    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:28:00.124641228Z","UUID":"6120e588-46ea-43e8-a6e9-257b2b185a85","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:28:00.747383 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:00.747355    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:28:00.747521 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:00.747398    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:28:00.776870 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:00.776843    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:28:00.777024 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:00.776886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:00.777024 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:00.776975    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:28:00.777142 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:00.777076    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:28:00.992625 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:00.992587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d" event={"ID":"15e524c0-e4ed-4337-8f70-06b943f1bf8b","Type":"ContainerStarted","Data":"1eb966fa24dfa62e4f5b7298e0a6356fc9f4c5d9dd006131494ed569db97a1f1"}
Sep 16 18:28:01.776724 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:01.776687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:28:01.776882 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:01.776817    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:28:01.997381 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:01.997315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d" event={"ID":"15e524c0-e4ed-4337-8f70-06b943f1bf8b","Type":"ContainerStarted","Data":"df28f604eace8fee82c2a3da84c6e10969c8aba037806e862794a2a7fe0b7e3b"}
Sep 16 18:28:02.000943 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:02.000912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs" event={"ID":"056dd948-046e-4d8d-bcd6-e96f225b93d5","Type":"ContainerStarted","Data":"32663bbf0d1b30721553b5adb75ed978e306e3017c7951c1902a438332a909b5"}
Sep 16 18:28:02.016814 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:02.016774    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbz6d" podStartSLOduration=4.127466528 podStartE2EDuration="24.016759858s" totalImagesPullingTime="19.88929333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:02.016100477 +0000 UTC m=+24.725339873" watchObservedRunningTime="2026-09-16 18:28:02.016759858 +0000 UTC m=+24.725999296"
Sep 16 18:28:02.171116 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:02.171033    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:28:02.171745 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:02.171725    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:28:02.777009 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:02.776978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:02.777009 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:02.777011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:28:02.777211 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:02.777101    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:28:02.777255 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:02.777228    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:28:03.777677 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:03.777457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:28:03.778047 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:03.777769    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:28:04.008863 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.008829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs" event={"ID":"056dd948-046e-4d8d-bcd6-e96f225b93d5","Type":"ContainerStarted","Data":"ae714dbae9e9b9f1c6b242b97b42ef5ad1379734217993419db29d2f482bd9e9"}
Sep 16 18:28:04.009098 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.009072    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:28:04.009295 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.009156    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:28:04.010939 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.010912    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef" containerID="8cee1ee58804830aed7b82a90ae67e50f68e1c68676db2da8977637d78cb9589" exitCode=0
Sep 16 18:28:04.011053 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.010948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtj68" event={"ID":"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef","Type":"ContainerDied","Data":"8cee1ee58804830aed7b82a90ae67e50f68e1c68676db2da8977637d78cb9589"}
Sep 16 18:28:04.024023 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.024004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:28:04.028698 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.028680    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:28:04.043699 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.043658    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs" podStartSLOduration=8.421593999 podStartE2EDuration="26.043645267s" totalImagesPullingTime="17.622051268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:04.042769342 +0000 UTC m=+26.752008746" watchObservedRunningTime="2026-09-16 18:28:04.043645267 +0000 UTC m=+26.752884659"
Sep 16 18:28:04.399933 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.399899    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:28:04.400077 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.400026    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:28:04.400441 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.400416    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mdgrv"
Sep 16 18:28:04.777079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.777058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:04.777183 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:04.777081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:28:04.777183 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:04.777158    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:28:04.777280 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:04.777263    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:28:05.015820 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:05.015751    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef" containerID="4d75bee9a7f7932869ac6e1a9e63c3cceef488c65b366ef9dfb0447fcb6d3bdf" exitCode=0
Sep 16 18:28:05.016428 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:05.015824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtj68" event={"ID":"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef","Type":"ContainerDied","Data":"4d75bee9a7f7932869ac6e1a9e63c3cceef488c65b366ef9dfb0447fcb6d3bdf"}
Sep 16 18:28:05.016428 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:05.015895    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:28:05.557654 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:05.557621    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rxwgv"]
Sep 16 18:28:05.557817 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:05.557748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:28:05.557872 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:05.557838    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:28:05.560406 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:05.560384    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-txjjd"]
Sep 16 18:28:05.560515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:05.560470    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:05.560561 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:05.560545    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:28:05.570712 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:05.570682    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jd7wp"]
Sep 16 18:28:05.570802 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:05.570790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:28:05.570938 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:05.570915    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:28:06.020468 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:06.020449    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:28:06.449489 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:06.449456    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:28:06.777531 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:06.777505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:06.777692 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:06.777596    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:28:07.024534 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:07.024500    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef" containerID="4c316d2d521bde1ac569493ab242dd7d3c73437cf91ef7b39c6171a6ef706bb1" exitCode=0
Sep 16 18:28:07.024927 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:07.024572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtj68" event={"ID":"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef","Type":"ContainerDied","Data":"4c316d2d521bde1ac569493ab242dd7d3c73437cf91ef7b39c6171a6ef706bb1"}
Sep 16 18:28:07.781188 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:07.781160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:28:07.781369 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:07.781271    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:28:07.781369 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:07.781321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:28:07.781484 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:07.781434    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:28:08.777723 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:08.777470    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:08.778095 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:08.777836    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:28:09.777449 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:09.777416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:28:09.777609 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:09.777588    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jd7wp" podUID="58e279fd-c891-476e-be69-1802a7262536"
Sep 16 18:28:09.777663 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:09.777597    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:28:09.777726 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:09.777692    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxwgv" podUID="17e3f8a4-0041-47e3-ade2-2744573aaa54"
Sep 16 18:28:10.777254 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:10.777219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:10.777834 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:10.777368    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txjjd" podUID="be53619d-4600-4740-b307-ab98564ff088"
Sep 16 18:28:11.047795 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.047703    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-16.ec2.internal" event="NodeReady"
Sep 16 18:28:11.047966 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.047862    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:28:11.084610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.084580    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66cbbd67bc-tzfsk"]
Sep 16 18:28:11.088428 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.088405    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.091371 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.091295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jnwzm\""
Sep 16 18:28:11.091371 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.091323    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:28:11.091528 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.091324    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:28:11.091528 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.091295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:28:11.100527 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.100493    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:28:11.102535 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.102507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66cbbd67bc-tzfsk"]
Sep 16 18:28:11.107494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.107468    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8wtgj"]
Sep 16 18:28:11.112504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.112478    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g8th6"]
Sep 16 18:28:11.112623 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.112598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.115000 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.114978    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sr6v8\""
Sep 16 18:28:11.115125 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.115105    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:28:11.115324 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.115307    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:28:11.118502 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.118479    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g8th6"
Sep 16 18:28:11.121447 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.120601    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8wtgj"]
Sep 16 18:28:11.121447 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.120970    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:28:11.121447 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.121175    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:28:11.121447 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.121398    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g75hp\""
Sep 16 18:28:11.121795 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.121648    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:28:11.133861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.133831    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g8th6"]
Sep 16 18:28:11.248648 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.248600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-certificates\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.248648 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.248649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-trusted-ca\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.248889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.248690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3fcd16-81ea-4a0a-8b03-f63208709c03-config-volume\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.248889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.248753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.248889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.248791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/810613e1-3990-43c3-b8f8-6bc0a1c2c81b-cert\") pod \"ingress-canary-g8th6\" (UID: \"810613e1-3990-43c3-b8f8-6bc0a1c2c81b\") " pod="openshift-ingress-canary/ingress-canary-g8th6"
Sep 16 18:28:11.248889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.248827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.248889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.248863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqsk\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-kube-api-access-bkqsk\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.248889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.248884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-image-registry-private-configuration\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.249191 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.248934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb2ht\" (UniqueName: \"kubernetes.io/projected/ef3fcd16-81ea-4a0a-8b03-f63208709c03-kube-api-access-wb2ht\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.249191 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.248966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-installation-pull-secrets\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.249191 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.249017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-ca-trust-extracted\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.249191 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.249041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef3fcd16-81ea-4a0a-8b03-f63208709c03-tmp-dir\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.249191 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.249093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-bound-sa-token\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.249191 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.249116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlcqm\" (UniqueName: \"kubernetes.io/projected/810613e1-3990-43c3-b8f8-6bc0a1c2c81b-kube-api-access-wlcqm\") pod \"ingress-canary-g8th6\" (UID: \"810613e1-3990-43c3-b8f8-6bc0a1c2c81b\") " pod="openshift-ingress-canary/ingress-canary-g8th6"
Sep 16 18:28:11.312901 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.312713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mdgrv_eb761fe6-5e93-438b-a415-2c50316fc733/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:11.349594 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-ca-trust-extracted\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.349594 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef3fcd16-81ea-4a0a-8b03-f63208709c03-tmp-dir\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.349823 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-bound-sa-token\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.349823 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlcqm\" (UniqueName: \"kubernetes.io/projected/810613e1-3990-43c3-b8f8-6bc0a1c2c81b-kube-api-access-wlcqm\") pod \"ingress-canary-g8th6\" (UID: \"810613e1-3990-43c3-b8f8-6bc0a1c2c81b\") " pod="openshift-ingress-canary/ingress-canary-g8th6"
Sep 16 18:28:11.349823 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-certificates\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.349823 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-trusted-ca\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.349823 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3fcd16-81ea-4a0a-8b03-f63208709c03-config-volume\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.350069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.350069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/810613e1-3990-43c3-b8f8-6bc0a1c2c81b-cert\") pod \"ingress-canary-g8th6\" (UID: \"810613e1-3990-43c3-b8f8-6bc0a1c2c81b\") " pod="openshift-ingress-canary/ingress-canary-g8th6"
Sep 16 18:28:11.350069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.350069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqsk\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-kube-api-access-bkqsk\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.350069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-image-registry-private-configuration\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.350069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb2ht\" (UniqueName: \"kubernetes.io/projected/ef3fcd16-81ea-4a0a-8b03-f63208709c03-kube-api-access-wb2ht\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.350069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.349974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-installation-pull-secrets\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.350069 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.350003    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:28:11.350069 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.350032    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66cbbd67bc-tzfsk: secret "image-registry-tls" not found
Sep 16 18:28:11.350069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.350030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef3fcd16-81ea-4a0a-8b03-f63208709c03-tmp-dir\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.350069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.350043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-ca-trust-extracted\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.350602 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.350100    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls podName:21bf4a5a-68bb-43ef-a991-57aee91b2fc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:11.850081244 +0000 UTC m=+34.559320619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls") pod "image-registry-66cbbd67bc-tzfsk" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4") : secret "image-registry-tls" not found
Sep 16 18:28:11.350694 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.350672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3fcd16-81ea-4a0a-8b03-f63208709c03-config-volume\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.350783 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.350767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-trusted-ca\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.350850 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.350838    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:28:11.350891 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.350887    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls podName:ef3fcd16-81ea-4a0a-8b03-f63208709c03 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:11.850874482 +0000 UTC m=+34.560113853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls") pod "dns-default-8wtgj" (UID: "ef3fcd16-81ea-4a0a-8b03-f63208709c03") : secret "dns-default-metrics-tls" not found
Sep 16 18:28:11.351188 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.351169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-certificates\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.354646 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.354626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-image-registry-private-configuration\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.354739 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.354649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/810613e1-3990-43c3-b8f8-6bc0a1c2c81b-cert\") pod \"ingress-canary-g8th6\" (UID: \"810613e1-3990-43c3-b8f8-6bc0a1c2c81b\") " pod="openshift-ingress-canary/ingress-canary-g8th6"
Sep 16 18:28:11.354739 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.354654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-installation-pull-secrets\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.360834 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.360803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqsk\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-kube-api-access-bkqsk\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.361396 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.361377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb2ht\" (UniqueName: \"kubernetes.io/projected/ef3fcd16-81ea-4a0a-8b03-f63208709c03-kube-api-access-wb2ht\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.362195 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.361950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-bound-sa-token\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.362195 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.361990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlcqm\" (UniqueName: \"kubernetes.io/projected/810613e1-3990-43c3-b8f8-6bc0a1c2c81b-kube-api-access-wlcqm\") pod \"ingress-canary-g8th6\" (UID: \"810613e1-3990-43c3-b8f8-6bc0a1c2c81b\") " pod="openshift-ingress-canary/ingress-canary-g8th6"
Sep 16 18:28:11.432656 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.432606    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g8th6"
Sep 16 18:28:11.450989 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.450949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:28:11.451174 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.451144    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:28:11.451233 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.451204    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs podName:58e279fd-c891-476e-be69-1802a7262536 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:43.451188192 +0000 UTC m=+66.160427586 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs") pod "network-metrics-daemon-jd7wp" (UID: "58e279fd-c891-476e-be69-1802a7262536") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:28:11.551904 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.551863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqps\" (UniqueName: \"kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps\") pod \"network-check-target-rxwgv\" (UID: \"17e3f8a4-0041-47e3-ade2-2744573aaa54\") " pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:28:11.552082 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.551985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:11.552082 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.552052    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:28:11.552082 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.552075    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:28:11.552240 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.552087    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rfqps for pod openshift-network-diagnostics/network-check-target-rxwgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:28:11.552240 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.552100    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:28:11.552240 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.552151    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps podName:17e3f8a4-0041-47e3-ade2-2744573aaa54 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:43.552131836 +0000 UTC m=+66.261371228 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfqps" (UniqueName: "kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps") pod "network-check-target-rxwgv" (UID: "17e3f8a4-0041-47e3-ade2-2744573aaa54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:28:11.552240 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.552175    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret podName:be53619d-4600-4740-b307-ab98564ff088 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:43.5521649 +0000 UTC m=+66.261404270 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret") pod "global-pull-secret-syncer-txjjd" (UID: "be53619d-4600-4740-b307-ab98564ff088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:28:11.603863 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.603759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mdgrv_eb761fe6-5e93-438b-a415-2c50316fc733/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:11.777451 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.777409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:28:11.777904 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.777409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:28:11.780396 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.780374    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:28:11.780502 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.780398    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:28:11.780502 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.780424    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pw4s4\""
Sep 16 18:28:11.780594 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.780518    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:28:11.780655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.780637    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p9wz2\""
Sep 16 18:28:11.854017 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.853936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:11.854178 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.854060    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:28:11.854178 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:11.854066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:11.854178 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.854102    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66cbbd67bc-tzfsk: secret "image-registry-tls" not found
Sep 16 18:28:11.854178 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.854172    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls podName:21bf4a5a-68bb-43ef-a991-57aee91b2fc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:12.854152366 +0000 UTC m=+35.563391736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls") pod "image-registry-66cbbd67bc-tzfsk" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4") : secret "image-registry-tls" not found
Sep 16 18:28:11.854402 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.854185    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:28:11.854402 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:11.854240    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls podName:ef3fcd16-81ea-4a0a-8b03-f63208709c03 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:12.854226736 +0000 UTC m=+35.563466123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls") pod "dns-default-8wtgj" (UID: "ef3fcd16-81ea-4a0a-8b03-f63208709c03") : secret "dns-default-metrics-tls" not found
Sep 16 18:28:12.450902 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:12.450695    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g8th6"]
Sep 16 18:28:12.455021 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:28:12.454986    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810613e1_3990_43c3_b8f8_6bc0a1c2c81b.slice/crio-33bf839992083a9bc5f5088b1bd3e043211effc8758add1e6e49e2baa0036a9c WatchSource:0}: Error finding container 33bf839992083a9bc5f5088b1bd3e043211effc8758add1e6e49e2baa0036a9c: Status 404 returned error can't find the container with id 33bf839992083a9bc5f5088b1bd3e043211effc8758add1e6e49e2baa0036a9c
Sep 16 18:28:12.776983 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:12.776955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:12.780108 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:12.780076    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:28:12.861037 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:12.861006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:12.861037 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:12.861040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:12.861292 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:12.861143    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:28:12.861292 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:12.861195    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls podName:ef3fcd16-81ea-4a0a-8b03-f63208709c03 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:14.861182035 +0000 UTC m=+37.570421404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls") pod "dns-default-8wtgj" (UID: "ef3fcd16-81ea-4a0a-8b03-f63208709c03") : secret "dns-default-metrics-tls" not found
Sep 16 18:28:12.861292 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:12.861195    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:28:12.861292 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:12.861218    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66cbbd67bc-tzfsk: secret "image-registry-tls" not found
Sep 16 18:28:12.861292 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:12.861274    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls podName:21bf4a5a-68bb-43ef-a991-57aee91b2fc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:14.861254664 +0000 UTC m=+37.570494054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls") pod "image-registry-66cbbd67bc-tzfsk" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4") : secret "image-registry-tls" not found
Sep 16 18:28:13.040753 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:13.040716    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef" containerID="5046f6d42fa0b089edfcbaf9354f461f46c8a02ec0071852fae7db458080e0ac" exitCode=0
Sep 16 18:28:13.040907 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:13.040786    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtj68" event={"ID":"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef","Type":"ContainerDied","Data":"5046f6d42fa0b089edfcbaf9354f461f46c8a02ec0071852fae7db458080e0ac"}
Sep 16 18:28:13.041957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:13.041932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g8th6" event={"ID":"810613e1-3990-43c3-b8f8-6bc0a1c2c81b","Type":"ContainerStarted","Data":"33bf839992083a9bc5f5088b1bd3e043211effc8758add1e6e49e2baa0036a9c"}
Sep 16 18:28:14.047523 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:14.047492    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef" containerID="f8e7077f6b6557e241d7f528b956f5214a9a1a32101dd2362cf91dcc2a108030" exitCode=0
Sep 16 18:28:14.048131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:14.047528    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtj68" event={"ID":"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef","Type":"ContainerDied","Data":"f8e7077f6b6557e241d7f528b956f5214a9a1a32101dd2362cf91dcc2a108030"}
Sep 16 18:28:14.877221 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:14.877185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:14.877405 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:14.877230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:14.877405 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:14.877308    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:28:14.877405 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:14.877306    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:28:14.877405 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:14.877391    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66cbbd67bc-tzfsk: secret "image-registry-tls" not found
Sep 16 18:28:14.877599 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:14.877368    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls podName:ef3fcd16-81ea-4a0a-8b03-f63208709c03 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:18.877355029 +0000 UTC m=+41.586594399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls") pod "dns-default-8wtgj" (UID: "ef3fcd16-81ea-4a0a-8b03-f63208709c03") : secret "dns-default-metrics-tls" not found
Sep 16 18:28:14.877599 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:14.877554    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls podName:21bf4a5a-68bb-43ef-a991-57aee91b2fc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:18.877536981 +0000 UTC m=+41.586776360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls") pod "image-registry-66cbbd67bc-tzfsk" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4") : secret "image-registry-tls" not found
Sep 16 18:28:15.052774 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:15.052738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtj68" event={"ID":"9f8b1fa8-e5b5-4101-b7ad-f46531ed7eef","Type":"ContainerStarted","Data":"96936ccb8e5ac3b26f6534012d9db94fc35986d67f5f93216f4a65455aa8aa08"}
Sep 16 18:28:15.077393 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:15.077349    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rtj68" podStartSLOduration=11.523989236 podStartE2EDuration="37.077323181s" totalImagesPullingTime="25.553333945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:15.075354079 +0000 UTC m=+37.784593465" watchObservedRunningTime="2026-09-16 18:28:15.077323181 +0000 UTC m=+37.786562628"
Sep 16 18:28:15.240668 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:15.240587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mdgrv_eb761fe6-5e93-438b-a415-2c50316fc733/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:15.329165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:15.329134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mdgrv_eb761fe6-5e93-438b-a415-2c50316fc733/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:15.436345 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:15.436300    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mdgrv_eb761fe6-5e93-438b-a415-2c50316fc733/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:15.535395 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:15.535361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mdgrv_eb761fe6-5e93-438b-a415-2c50316fc733/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:16.056704 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:16.056666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g8th6" event={"ID":"810613e1-3990-43c3-b8f8-6bc0a1c2c81b","Type":"ContainerStarted","Data":"15b4738a569c068ccac0471f544ec648b424a3cd24ab703ba68ee3412e34054a"}
Sep 16 18:28:16.073480 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:16.073431    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g8th6" podStartSLOduration=2.288314878 podStartE2EDuration="5.073418233s" totalImagesPullingTime="2.785103355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:16.072923693 +0000 UTC m=+38.782163084" watchObservedRunningTime="2026-09-16 18:28:16.073418233 +0000 UTC m=+38.782657624"
Sep 16 18:28:18.907472 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:18.907424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:18.907472 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:18.907470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:18.907920 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:18.907568    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:28:18.907920 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:18.907589    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66cbbd67bc-tzfsk: secret "image-registry-tls" not found
Sep 16 18:28:18.907920 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:18.907611    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:28:18.907920 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:18.907641    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls podName:21bf4a5a-68bb-43ef-a991-57aee91b2fc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:26.907626133 +0000 UTC m=+49.616865503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls") pod "image-registry-66cbbd67bc-tzfsk" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4") : secret "image-registry-tls" not found
Sep 16 18:28:18.907920 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:18.907674    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls podName:ef3fcd16-81ea-4a0a-8b03-f63208709c03 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:26.907658222 +0000 UTC m=+49.616897592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls") pod "dns-default-8wtgj" (UID: "ef3fcd16-81ea-4a0a-8b03-f63208709c03") : secret "dns-default-metrics-tls" not found
Sep 16 18:28:26.967098 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:26.967052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:26.967098 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:26.967093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:26.967620 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:26.967205    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:28:26.967620 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:26.967215    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:28:26.967620 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:26.967236    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66cbbd67bc-tzfsk: secret "image-registry-tls" not found
Sep 16 18:28:26.967620 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:26.967262    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls podName:ef3fcd16-81ea-4a0a-8b03-f63208709c03 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:42.967245532 +0000 UTC m=+65.676484905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls") pod "dns-default-8wtgj" (UID: "ef3fcd16-81ea-4a0a-8b03-f63208709c03") : secret "dns-default-metrics-tls" not found
Sep 16 18:28:26.967620 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:26.967290    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls podName:21bf4a5a-68bb-43ef-a991-57aee91b2fc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:42.967276363 +0000 UTC m=+65.676515734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls") pod "image-registry-66cbbd67bc-tzfsk" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4") : secret "image-registry-tls" not found
Sep 16 18:28:31.063555 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.063515    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bbkt2"]
Sep 16 18:28:31.065189 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.065171    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.068165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.068131    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:28:31.068297 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.068143    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mv7x7\""
Sep 16 18:28:31.069599 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.069564    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:28:31.070671 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.070644    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:28:31.070931 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.070909    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:28:31.072989 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.072969    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:28:31.084669 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.084643    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bbkt2"]
Sep 16 18:28:31.153105 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.153066    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"]
Sep 16 18:28:31.155233 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.155207    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"]
Sep 16 18:28:31.155369 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.155353    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"
Sep 16 18:28:31.156847 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.156827    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:31.160958 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.160934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:28:31.166357 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.166340    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:28:31.167080 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.167005    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:28:31.167841 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.167826    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:28:31.167938 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.167857    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:28:31.168365 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.168348    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:28:31.168431 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.168363    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-c8k7m\""
Sep 16 18:28:31.168431 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.168363    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wb6kz\""
Sep 16 18:28:31.176445 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.176425    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"]
Sep 16 18:28:31.177046 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.177026    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:28:31.180296 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.180275    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"]
Sep 16 18:28:31.198552 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.198529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4222a382-cb85-4cd2-b8ed-0dd10fa26702-trusted-ca\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.198660 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.198578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4222a382-cb85-4cd2-b8ed-0dd10fa26702-serving-cert\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.198660 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.198626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96zwb\" (UniqueName: \"kubernetes.io/projected/4222a382-cb85-4cd2-b8ed-0dd10fa26702-kube-api-access-96zwb\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.198725 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.198663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4222a382-cb85-4cd2-b8ed-0dd10fa26702-config\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.299538 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.299504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96zwb\" (UniqueName: \"kubernetes.io/projected/4222a382-cb85-4cd2-b8ed-0dd10fa26702-kube-api-access-96zwb\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.299790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.299545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d46e260-7277-4f07-8087-6b3d867252af-serving-cert\") pod \"service-ca-operator-cf7c776b7-npd24\" (UID: \"4d46e260-7277-4f07-8087-6b3d867252af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"
Sep 16 18:28:31.299790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.299572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d46e260-7277-4f07-8087-6b3d867252af-config\") pod \"service-ca-operator-cf7c776b7-npd24\" (UID: \"4d46e260-7277-4f07-8087-6b3d867252af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"
Sep 16 18:28:31.299790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.299692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mw5n9\" (UID: \"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:31.299790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.299746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86fqq\" (UniqueName: \"kubernetes.io/projected/4d46e260-7277-4f07-8087-6b3d867252af-kube-api-access-86fqq\") pod \"service-ca-operator-cf7c776b7-npd24\" (UID: \"4d46e260-7277-4f07-8087-6b3d867252af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"
Sep 16 18:28:31.299790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.299772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppdz\" (UniqueName: \"kubernetes.io/projected/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-kube-api-access-wppdz\") pod \"cluster-samples-operator-86499c89c4-mw5n9\" (UID: \"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:31.300019 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.299807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4222a382-cb85-4cd2-b8ed-0dd10fa26702-config\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.300019 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.299952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4222a382-cb85-4cd2-b8ed-0dd10fa26702-trusted-ca\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.300019 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.300002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4222a382-cb85-4cd2-b8ed-0dd10fa26702-serving-cert\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.300426 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.300407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4222a382-cb85-4cd2-b8ed-0dd10fa26702-config\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.300719 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.300697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4222a382-cb85-4cd2-b8ed-0dd10fa26702-trusted-ca\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.303389 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.303368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4222a382-cb85-4cd2-b8ed-0dd10fa26702-serving-cert\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.307970 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.307950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96zwb\" (UniqueName: \"kubernetes.io/projected/4222a382-cb85-4cd2-b8ed-0dd10fa26702-kube-api-access-96zwb\") pod \"console-operator-696884fb9c-bbkt2\" (UID: \"4222a382-cb85-4cd2-b8ed-0dd10fa26702\") " pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.375925 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.375845    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:31.401412 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.401381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d46e260-7277-4f07-8087-6b3d867252af-serving-cert\") pod \"service-ca-operator-cf7c776b7-npd24\" (UID: \"4d46e260-7277-4f07-8087-6b3d867252af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"
Sep 16 18:28:31.401542 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.401426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d46e260-7277-4f07-8087-6b3d867252af-config\") pod \"service-ca-operator-cf7c776b7-npd24\" (UID: \"4d46e260-7277-4f07-8087-6b3d867252af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"
Sep 16 18:28:31.401542 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.401447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mw5n9\" (UID: \"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:31.401542 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.401482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86fqq\" (UniqueName: \"kubernetes.io/projected/4d46e260-7277-4f07-8087-6b3d867252af-kube-api-access-86fqq\") pod \"service-ca-operator-cf7c776b7-npd24\" (UID: \"4d46e260-7277-4f07-8087-6b3d867252af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"
Sep 16 18:28:31.401542 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.401505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppdz\" (UniqueName: \"kubernetes.io/projected/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-kube-api-access-wppdz\") pod \"cluster-samples-operator-86499c89c4-mw5n9\" (UID: \"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:31.401704 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:31.401625    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:28:31.401742 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:31.401718    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls podName:8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:31.901697054 +0000 UTC m=+54.610936452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-mw5n9" (UID: "8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04") : secret "samples-operator-tls" not found
Sep 16 18:28:31.402096 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.402076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d46e260-7277-4f07-8087-6b3d867252af-config\") pod \"service-ca-operator-cf7c776b7-npd24\" (UID: \"4d46e260-7277-4f07-8087-6b3d867252af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"
Sep 16 18:28:31.403648 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.403626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d46e260-7277-4f07-8087-6b3d867252af-serving-cert\") pod \"service-ca-operator-cf7c776b7-npd24\" (UID: \"4d46e260-7277-4f07-8087-6b3d867252af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"
Sep 16 18:28:31.414722 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.414699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86fqq\" (UniqueName: \"kubernetes.io/projected/4d46e260-7277-4f07-8087-6b3d867252af-kube-api-access-86fqq\") pod \"service-ca-operator-cf7c776b7-npd24\" (UID: \"4d46e260-7277-4f07-8087-6b3d867252af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"
Sep 16 18:28:31.416083 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.416064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppdz\" (UniqueName: \"kubernetes.io/projected/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-kube-api-access-wppdz\") pod \"cluster-samples-operator-86499c89c4-mw5n9\" (UID: \"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:31.465626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.465587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"
Sep 16 18:28:31.645203 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.645132    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bbkt2"]
Sep 16 18:28:31.648545 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:28:31.648516    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4222a382_cb85_4cd2_b8ed_0dd10fa26702.slice/crio-bb5ea9a4c5c1676ad4952c77c5f828f2800782d055028ae295d6db355e6f41af WatchSource:0}: Error finding container bb5ea9a4c5c1676ad4952c77c5f828f2800782d055028ae295d6db355e6f41af: Status 404 returned error can't find the container with id bb5ea9a4c5c1676ad4952c77c5f828f2800782d055028ae295d6db355e6f41af
Sep 16 18:28:31.732001 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.731968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24"]
Sep 16 18:28:31.735098 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:28:31.735072    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d46e260_7277_4f07_8087_6b3d867252af.slice/crio-a646431c6b52ffb4b4598125ef7af7eb17d54a75c3261b05acce4908f1bb2c21 WatchSource:0}: Error finding container a646431c6b52ffb4b4598125ef7af7eb17d54a75c3261b05acce4908f1bb2c21: Status 404 returned error can't find the container with id a646431c6b52ffb4b4598125ef7af7eb17d54a75c3261b05acce4908f1bb2c21
Sep 16 18:28:31.906060 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:31.905980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mw5n9\" (UID: \"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:31.906197 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:31.906133    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:28:31.906235 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:31.906196    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls podName:8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:32.906178285 +0000 UTC m=+55.615417658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-mw5n9" (UID: "8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04") : secret "samples-operator-tls" not found
Sep 16 18:28:32.092591 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:32.092554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2" event={"ID":"4222a382-cb85-4cd2-b8ed-0dd10fa26702","Type":"ContainerStarted","Data":"bb5ea9a4c5c1676ad4952c77c5f828f2800782d055028ae295d6db355e6f41af"}
Sep 16 18:28:32.093727 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:32.093701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24" event={"ID":"4d46e260-7277-4f07-8087-6b3d867252af","Type":"ContainerStarted","Data":"a646431c6b52ffb4b4598125ef7af7eb17d54a75c3261b05acce4908f1bb2c21"}
Sep 16 18:28:32.913992 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:32.913954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mw5n9\" (UID: \"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:32.914177 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:32.914104    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:28:32.914177 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:32.914173    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls podName:8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:34.91415441 +0000 UTC m=+57.623393780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-mw5n9" (UID: "8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04") : secret "samples-operator-tls" not found
Sep 16 18:28:34.927474 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:34.927380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mw5n9\" (UID: \"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:34.927816 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:34.927531    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:28:34.927816 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:34.927598    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls podName:8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:38.927579766 +0000 UTC m=+61.636819155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-mw5n9" (UID: "8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04") : secret "samples-operator-tls" not found
Sep 16 18:28:35.103394 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:35.103365    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/0.log"
Sep 16 18:28:35.103572 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:35.103410    2580 generic.go:356] "Generic (PLEG): container finished" podID="4222a382-cb85-4cd2-b8ed-0dd10fa26702" containerID="e37a87d21f4eed1f85f441eaca4bd0dc43e2231453cc48457ade8178a9187211" exitCode=255
Sep 16 18:28:35.103572 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:35.103490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2" event={"ID":"4222a382-cb85-4cd2-b8ed-0dd10fa26702","Type":"ContainerDied","Data":"e37a87d21f4eed1f85f441eaca4bd0dc43e2231453cc48457ade8178a9187211"}
Sep 16 18:28:35.103700 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:35.103675    2580 scope.go:117] "RemoveContainer" containerID="e37a87d21f4eed1f85f441eaca4bd0dc43e2231453cc48457ade8178a9187211"
Sep 16 18:28:35.105034 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:35.105008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24" event={"ID":"4d46e260-7277-4f07-8087-6b3d867252af","Type":"ContainerStarted","Data":"460c28fd41b79da803228b32037bd71dbd59c1d5cda62387a18ac65581224fc2"}
Sep 16 18:28:35.146257 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:35.146204    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24" podStartSLOduration=1.306830701 podStartE2EDuration="4.146184175s" totalImagesPullingTime="2.839353474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:35.145352178 +0000 UTC m=+57.854591594" watchObservedRunningTime="2026-09-16 18:28:35.146184175 +0000 UTC m=+57.855423568"
Sep 16 18:28:36.109450 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:36.109414    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/1.log"
Sep 16 18:28:36.109867 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:36.109852    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/0.log"
Sep 16 18:28:36.109919 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:36.109891    2580 generic.go:356] "Generic (PLEG): container finished" podID="4222a382-cb85-4cd2-b8ed-0dd10fa26702" containerID="80dc3fb87c0bbc25b17f3363434d7931d3d042bb75c50c1ebea263b348e24c68" exitCode=255
Sep 16 18:28:36.110012 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:36.109984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2" event={"ID":"4222a382-cb85-4cd2-b8ed-0dd10fa26702","Type":"ContainerDied","Data":"80dc3fb87c0bbc25b17f3363434d7931d3d042bb75c50c1ebea263b348e24c68"}
Sep 16 18:28:36.110085 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:36.110073    2580 scope.go:117] "RemoveContainer" containerID="e37a87d21f4eed1f85f441eaca4bd0dc43e2231453cc48457ade8178a9187211"
Sep 16 18:28:36.110283 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:36.110265    2580 scope.go:117] "RemoveContainer" containerID="80dc3fb87c0bbc25b17f3363434d7931d3d042bb75c50c1ebea263b348e24c68"
Sep 16 18:28:36.110464 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:36.110443    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bbkt2_openshift-console-operator(4222a382-cb85-4cd2-b8ed-0dd10fa26702)\"" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2" podUID="4222a382-cb85-4cd2-b8ed-0dd10fa26702"
Sep 16 18:28:37.034599 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.034574    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b69vs"
Sep 16 18:28:37.113594 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.113569    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/1.log"
Sep 16 18:28:37.113972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.113867    2580 scope.go:117] "RemoveContainer" containerID="80dc3fb87c0bbc25b17f3363434d7931d3d042bb75c50c1ebea263b348e24c68"
Sep 16 18:28:37.114028 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:37.114009    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bbkt2_openshift-console-operator(4222a382-cb85-4cd2-b8ed-0dd10fa26702)\"" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2" podUID="4222a382-cb85-4cd2-b8ed-0dd10fa26702"
Sep 16 18:28:37.570696 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.570662    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2jvx9"]
Sep 16 18:28:37.572842 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.572825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.575392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.575370    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:28:37.575514 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.575496    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:28:37.575721 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.575704    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:28:37.575784 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.575711    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:28:37.575784 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.575756    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p5rq9\""
Sep 16 18:28:37.585855 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.585833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2jvx9"]
Sep 16 18:28:37.648797 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.648762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8017534a-2ecf-4a17-87cc-2395a913d221-data-volume\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.648959 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.648807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.648959 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.648886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8017534a-2ecf-4a17-87cc-2395a913d221-crio-socket\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.649048 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.648959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkg79\" (UniqueName: \"kubernetes.io/projected/8017534a-2ecf-4a17-87cc-2395a913d221-kube-api-access-mkg79\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.649048 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.648998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.749269 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.749239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8017534a-2ecf-4a17-87cc-2395a913d221-data-volume\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.749269 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.749270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.749504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.749321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8017534a-2ecf-4a17-87cc-2395a913d221-crio-socket\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.749504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.749394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkg79\" (UniqueName: \"kubernetes.io/projected/8017534a-2ecf-4a17-87cc-2395a913d221-kube-api-access-mkg79\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.749504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.749486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8017534a-2ecf-4a17-87cc-2395a913d221-crio-socket\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.749665 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.749535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.749665 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.749555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8017534a-2ecf-4a17-87cc-2395a913d221-data-volume\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:37.752131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.752116    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:28:37.752186 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.752170    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:28:37.758225 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.758211    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:28:37.760290 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:37.760275    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:28:37.760349 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:37.760290    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:28:37.760385 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:37.760352    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls podName:8017534a-2ecf-4a17-87cc-2395a913d221 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:38.260325136 +0000 UTC m=+60.969564507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2jvx9" (UID: "8017534a-2ecf-4a17-87cc-2395a913d221") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:28:37.760385 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:37.760367    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm podName:8017534a-2ecf-4a17-87cc-2395a913d221 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:38.260359992 +0000 UTC m=+60.969599362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2jvx9" (UID: "8017534a-2ecf-4a17-87cc-2395a913d221") : configmap "kube-rbac-proxy" not found
Sep 16 18:28:37.768767 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.768749    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:28:37.778820 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:37.778783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkg79\" (UniqueName: \"kubernetes.io/projected/8017534a-2ecf-4a17-87cc-2395a913d221-kube-api-access-mkg79\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:38.355212 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:38.355170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:38.355674 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:38.355240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:38.355674 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:38.355303    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:28:38.355674 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:38.355407    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm podName:8017534a-2ecf-4a17-87cc-2395a913d221 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:39.355392621 +0000 UTC m=+62.064631991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2jvx9" (UID: "8017534a-2ecf-4a17-87cc-2395a913d221") : configmap "kube-rbac-proxy" not found
Sep 16 18:28:38.355674 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:38.355434    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:28:38.355674 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:38.355495    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls podName:8017534a-2ecf-4a17-87cc-2395a913d221 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:39.35548153 +0000 UTC m=+62.064720913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2jvx9" (UID: "8017534a-2ecf-4a17-87cc-2395a913d221") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:28:38.959298 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:38.959270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mw5n9\" (UID: \"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:38.959509 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:38.959419    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:28:38.959509 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:38.959492    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls podName:8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:46.959475079 +0000 UTC m=+69.668714461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-mw5n9" (UID: "8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04") : secret "samples-operator-tls" not found
Sep 16 18:28:39.362736 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:39.362702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:39.363105 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:39.362757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:39.363105 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:39.362842    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:28:39.363105 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:39.362864    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:28:39.363105 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:39.362921    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm podName:8017534a-2ecf-4a17-87cc-2395a913d221 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:41.362901474 +0000 UTC m=+64.072140857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2jvx9" (UID: "8017534a-2ecf-4a17-87cc-2395a913d221") : configmap "kube-rbac-proxy" not found
Sep 16 18:28:39.363105 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:39.362940    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls podName:8017534a-2ecf-4a17-87cc-2395a913d221 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:41.362931943 +0000 UTC m=+64.072171335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2jvx9" (UID: "8017534a-2ecf-4a17-87cc-2395a913d221") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:28:40.302217 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:40.302184    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r5z5r_7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46/dns-node-resolver/0.log"
Sep 16 18:28:41.376280 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:41.376243    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:41.376785 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:41.376305    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:28:41.376785 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:41.376726    2580 scope.go:117] "RemoveContainer" containerID="80dc3fb87c0bbc25b17f3363434d7931d3d042bb75c50c1ebea263b348e24c68"
Sep 16 18:28:41.376938 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:41.376914    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bbkt2_openshift-console-operator(4222a382-cb85-4cd2-b8ed-0dd10fa26702)\"" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2" podUID="4222a382-cb85-4cd2-b8ed-0dd10fa26702"
Sep 16 18:28:41.377894 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:41.377870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:41.377982 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:41.377917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:41.378019 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:41.377985    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:28:41.378052 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:41.378035    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:28:41.378088 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:41.378039    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm podName:8017534a-2ecf-4a17-87cc-2395a913d221 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:45.378024283 +0000 UTC m=+68.087263653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2jvx9" (UID: "8017534a-2ecf-4a17-87cc-2395a913d221") : configmap "kube-rbac-proxy" not found
Sep 16 18:28:41.378088 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:41.378083    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls podName:8017534a-2ecf-4a17-87cc-2395a913d221 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:45.378071405 +0000 UTC m=+68.087310776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2jvx9" (UID: "8017534a-2ecf-4a17-87cc-2395a913d221") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:28:41.501815 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:41.501787    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bfn66_cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2/node-ca/0.log"
Sep 16 18:28:42.302135 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:42.302102    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g8th6_810613e1-3990-43c3-b8f8-6bc0a1c2c81b/serve-healthcheck-canary/0.log"
Sep 16 18:28:42.991995 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:42.991953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:28:42.991995 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:42.991994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:28:42.992459 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:42.992101    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:28:42.992459 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:42.992151    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls podName:ef3fcd16-81ea-4a0a-8b03-f63208709c03 nodeName:}" failed. No retries permitted until 2026-09-16 18:29:14.992138284 +0000 UTC m=+97.701377654 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls") pod "dns-default-8wtgj" (UID: "ef3fcd16-81ea-4a0a-8b03-f63208709c03") : secret "dns-default-metrics-tls" not found
Sep 16 18:28:42.992459 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:42.992204    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:28:42.992459 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:42.992222    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66cbbd67bc-tzfsk: secret "image-registry-tls" not found
Sep 16 18:28:42.992459 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:42.992269    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls podName:21bf4a5a-68bb-43ef-a991-57aee91b2fc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:29:14.992256185 +0000 UTC m=+97.701495554 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls") pod "image-registry-66cbbd67bc-tzfsk" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4") : secret "image-registry-tls" not found
Sep 16 18:28:43.496410 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.496381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:28:43.498939 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.498920    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:28:43.506906 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:43.506885    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:28:43.506951 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:43.506939    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs podName:58e279fd-c891-476e-be69-1802a7262536 nodeName:}" failed. No retries permitted until 2026-09-16 18:29:47.506924897 +0000 UTC m=+130.216164266 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs") pod "network-metrics-daemon-jd7wp" (UID: "58e279fd-c891-476e-be69-1802a7262536") : secret "metrics-daemon-secret" not found
Sep 16 18:28:43.596919 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.596882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:43.597064 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.596928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqps\" (UniqueName: \"kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps\") pod \"network-check-target-rxwgv\" (UID: \"17e3f8a4-0041-47e3-ade2-2744573aaa54\") " pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:28:43.599535 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.599513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:28:43.599581 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.599544    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:28:43.609587 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.609555    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:28:43.609827 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.609804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be53619d-4600-4740-b307-ab98564ff088-original-pull-secret\") pod \"global-pull-secret-syncer-txjjd\" (UID: \"be53619d-4600-4740-b307-ab98564ff088\") " pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:43.620115 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.620092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfqps\" (UniqueName: \"kubernetes.io/projected/17e3f8a4-0041-47e3-ade2-2744573aaa54-kube-api-access-rfqps\") pod \"network-check-target-rxwgv\" (UID: \"17e3f8a4-0041-47e3-ade2-2744573aaa54\") " pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:28:43.688743 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.688705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txjjd"
Sep 16 18:28:43.891660 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.891627    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pw4s4\""
Sep 16 18:28:43.899867 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.899850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:28:43.961430 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:43.961356    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-txjjd"]
Sep 16 18:28:43.964346 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:28:43.964304    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe53619d_4600_4740_b307_ab98564ff088.slice/crio-e6d1fb77f0fdefdab68d6a2d38c48d4a152edd819e1c7e7dad312c6b38b77293 WatchSource:0}: Error finding container e6d1fb77f0fdefdab68d6a2d38c48d4a152edd819e1c7e7dad312c6b38b77293: Status 404 returned error can't find the container with id e6d1fb77f0fdefdab68d6a2d38c48d4a152edd819e1c7e7dad312c6b38b77293
Sep 16 18:28:44.130267 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:44.130229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-txjjd" event={"ID":"be53619d-4600-4740-b307-ab98564ff088","Type":"ContainerStarted","Data":"e6d1fb77f0fdefdab68d6a2d38c48d4a152edd819e1c7e7dad312c6b38b77293"}
Sep 16 18:28:44.164498 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:44.164428    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rxwgv"]
Sep 16 18:28:44.167278 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:28:44.167257    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e3f8a4_0041_47e3_ade2_2744573aaa54.slice/crio-b8df2e63199c0264ff6493c6d7f7f1f5e803834bf8ca1118af2aaa5cff06219f WatchSource:0}: Error finding container b8df2e63199c0264ff6493c6d7f7f1f5e803834bf8ca1118af2aaa5cff06219f: Status 404 returned error can't find the container with id b8df2e63199c0264ff6493c6d7f7f1f5e803834bf8ca1118af2aaa5cff06219f
Sep 16 18:28:45.134177 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:45.134145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rxwgv" event={"ID":"17e3f8a4-0041-47e3-ade2-2744573aaa54","Type":"ContainerStarted","Data":"b8df2e63199c0264ff6493c6d7f7f1f5e803834bf8ca1118af2aaa5cff06219f"}
Sep 16 18:28:45.411118 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:45.411033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:45.411281 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:45.411162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:45.411281 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:45.411216    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:28:45.411281 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:45.411277    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:28:45.411469 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:45.411345    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls podName:8017534a-2ecf-4a17-87cc-2395a913d221 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:53.411304527 +0000 UTC m=+76.120543911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2jvx9" (UID: "8017534a-2ecf-4a17-87cc-2395a913d221") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:28:45.411469 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:45.411368    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm podName:8017534a-2ecf-4a17-87cc-2395a913d221 nodeName:}" failed. No retries permitted until 2026-09-16 18:28:53.411356495 +0000 UTC m=+76.120595879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2jvx9" (UID: "8017534a-2ecf-4a17-87cc-2395a913d221") : configmap "kube-rbac-proxy" not found
Sep 16 18:28:47.024392 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:47.024355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mw5n9\" (UID: \"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:47.027304 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:47.027275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mw5n9\" (UID: \"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:47.072725 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:47.072688    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wb6kz\""
Sep 16 18:28:47.080816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:47.080778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"
Sep 16 18:28:48.871311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:48.871226    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9"]
Sep 16 18:28:49.145646 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:49.145560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-txjjd" event={"ID":"be53619d-4600-4740-b307-ab98564ff088","Type":"ContainerStarted","Data":"7d625fb3cf8e2cd7be63ee81f2a3b0c90d4f3b40328cafbb380f83e56649cfed"}
Sep 16 18:28:49.147044 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:49.147020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rxwgv" event={"ID":"17e3f8a4-0041-47e3-ade2-2744573aaa54","Type":"ContainerStarted","Data":"75ce14092f43544e784d5e9d526cdbfdcb5db34845cc3ad71bfefdb2d9dacd60"}
Sep 16 18:28:49.147186 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:49.147079    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:28:49.148138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:49.148118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9" event={"ID":"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04","Type":"ContainerStarted","Data":"0b8014d22057f091475448bf8cc2ee80e19d39e124f3057364f06d408a025f51"}
Sep 16 18:28:49.163488 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:49.163441    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-txjjd" podStartSLOduration=66.511655832 podStartE2EDuration="1m11.163426685s" totalImagesPullingTime="4.651770853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:49.163058073 +0000 UTC m=+71.872297466" watchObservedRunningTime="2026-09-16 18:28:49.163426685 +0000 UTC m=+71.872666076"
Sep 16 18:28:49.178886 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:49.178844    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rxwgv" podStartSLOduration=66.745886364 podStartE2EDuration="1m11.17883094s" totalImagesPullingTime="4.432944576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:49.178506912 +0000 UTC m=+71.887746307" watchObservedRunningTime="2026-09-16 18:28:49.17883094 +0000 UTC m=+71.888070333"
Sep 16 18:28:51.156015 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:51.155981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9" event={"ID":"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04","Type":"ContainerStarted","Data":"52bbf0a839b7f6b2035ea511dfc1c1afb6c113895777dda37682d8d187c42409"}
Sep 16 18:28:51.156397 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:51.156035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9" event={"ID":"8e1b5962-b5c6-4ccc-b9e5-dc2d45036d04","Type":"ContainerStarted","Data":"bc3e28e8088fde488eeb9328b237f2070e54696f012b86a8ede9130e3c6289c8"}
Sep 16 18:28:51.174537 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:51.174488    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mw5n9" podStartSLOduration=18.597448652 podStartE2EDuration="20.174471758s" totalImagesPullingTime="1.577023106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:51.173426586 +0000 UTC m=+73.882665977" watchObservedRunningTime="2026-09-16 18:28:51.174471758 +0000 UTC m=+73.883711149"
Sep 16 18:28:53.474375 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:53.474313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:53.474862 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:53.474409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:53.474913 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:53.474887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8017534a-2ecf-4a17-87cc-2395a913d221-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:53.476546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:53.476526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8017534a-2ecf-4a17-87cc-2395a913d221-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jvx9\" (UID: \"8017534a-2ecf-4a17-87cc-2395a913d221\") " pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:53.484091 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:53.484071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p5rq9\""
Sep 16 18:28:53.491893 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:53.491879    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2jvx9"
Sep 16 18:28:53.761217 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:53.761145    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2jvx9"]
Sep 16 18:28:53.763957 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:28:53.763931    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8017534a_2ecf_4a17_87cc_2395a913d221.slice/crio-1dd3f9676b7d4c32cdc7d817342aca2de0ccfb02c13b3ad790ff7fd1317394d4 WatchSource:0}: Error finding container 1dd3f9676b7d4c32cdc7d817342aca2de0ccfb02c13b3ad790ff7fd1317394d4: Status 404 returned error can't find the container with id 1dd3f9676b7d4c32cdc7d817342aca2de0ccfb02c13b3ad790ff7fd1317394d4
Sep 16 18:28:53.777274 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:53.777254    2580 scope.go:117] "RemoveContainer" containerID="80dc3fb87c0bbc25b17f3363434d7931d3d042bb75c50c1ebea263b348e24c68"
Sep 16 18:28:54.170967 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:54.170938    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:28:54.171383 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:54.171366    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/1.log"
Sep 16 18:28:54.171462 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:54.171399    2580 generic.go:356] "Generic (PLEG): container finished" podID="4222a382-cb85-4cd2-b8ed-0dd10fa26702" containerID="23f0c8ccfa4e99e0210ceed89f008e3203671bcabb5bb990370e3a990a415d4c" exitCode=255
Sep 16 18:28:54.171510 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:54.171465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2" event={"ID":"4222a382-cb85-4cd2-b8ed-0dd10fa26702","Type":"ContainerDied","Data":"23f0c8ccfa4e99e0210ceed89f008e3203671bcabb5bb990370e3a990a415d4c"}
Sep 16 18:28:54.171558 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:54.171513    2580 scope.go:117] "RemoveContainer" containerID="80dc3fb87c0bbc25b17f3363434d7931d3d042bb75c50c1ebea263b348e24c68"
Sep 16 18:28:54.171862 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:54.171834    2580 scope.go:117] "RemoveContainer" containerID="23f0c8ccfa4e99e0210ceed89f008e3203671bcabb5bb990370e3a990a415d4c"
Sep 16 18:28:54.172113 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:28:54.172013    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-bbkt2_openshift-console-operator(4222a382-cb85-4cd2-b8ed-0dd10fa26702)\"" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2" podUID="4222a382-cb85-4cd2-b8ed-0dd10fa26702"
Sep 16 18:28:54.173049 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:54.173021    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2jvx9" event={"ID":"8017534a-2ecf-4a17-87cc-2395a913d221","Type":"ContainerStarted","Data":"20b396d82a16c17be2cdff2e1c19baf024d4270ef51930995415f36bb8ac729b"}
Sep 16 18:28:54.173139 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:54.173078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2jvx9" event={"ID":"8017534a-2ecf-4a17-87cc-2395a913d221","Type":"ContainerStarted","Data":"1dd3f9676b7d4c32cdc7d817342aca2de0ccfb02c13b3ad790ff7fd1317394d4"}
Sep 16 18:28:55.179878 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:55.179848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:28:55.181518 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:55.181494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2jvx9" event={"ID":"8017534a-2ecf-4a17-87cc-2395a913d221","Type":"ContainerStarted","Data":"d10036eb0830e793c8bd98ff5c2ac7e8c10f4c1c83047d13b83b2876a6bff521"}
Sep 16 18:28:57.189776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:57.189734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2jvx9" event={"ID":"8017534a-2ecf-4a17-87cc-2395a913d221","Type":"ContainerStarted","Data":"64c1f6dfaf31d3dd8ad8544145a9ac419a95b231283e8cbf3346da5317841a74"}
Sep 16 18:28:57.226299 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:57.226249    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2jvx9" podStartSLOduration=17.900239218 podStartE2EDuration="20.226234843s" totalImagesPullingTime="2.325995625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:28:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:57.225238224 +0000 UTC m=+79.934477615" watchObservedRunningTime="2026-09-16 18:28:57.226234843 +0000 UTC m=+79.935474235"
Sep 16 18:28:59.492438 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.492404    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q"]
Sep 16 18:28:59.496767 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.496749    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q"
Sep 16 18:28:59.501278 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.501255    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:28:59.501278 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.501269    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:28:59.504213 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.504190    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:28:59.504351 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.504190    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:28:59.504351 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.504299    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lm7ts\""
Sep 16 18:28:59.509764 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.509740    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q"]
Sep 16 18:28:59.591482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.591452    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d"]
Sep 16 18:28:59.594270 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.594247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d"
Sep 16 18:28:59.596698 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.596677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6rlwd\""
Sep 16 18:28:59.596810 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.596779    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:28:59.597036 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.597002    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:28:59.609341 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.609306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d"]
Sep 16 18:28:59.623566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.623543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2588625-c9a6-40bf-a6b8-e55b26dfa2cf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7887965c76-tzl7q\" (UID: \"c2588625-c9a6-40bf-a6b8-e55b26dfa2cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q"
Sep 16 18:28:59.623685 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.623576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjvc\" (UniqueName: \"kubernetes.io/projected/c2588625-c9a6-40bf-a6b8-e55b26dfa2cf-kube-api-access-pdjvc\") pod \"managed-serviceaccount-addon-agent-7887965c76-tzl7q\" (UID: \"c2588625-c9a6-40bf-a6b8-e55b26dfa2cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q"
Sep 16 18:28:59.724848 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.724812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b7e2fe2-787d-4a41-994b-d833daae8743-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2d65d\" (UID: \"9b7e2fe2-787d-4a41-994b-d833daae8743\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d"
Sep 16 18:28:59.724848 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.724850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b7e2fe2-787d-4a41-994b-d833daae8743-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2d65d\" (UID: \"9b7e2fe2-787d-4a41-994b-d833daae8743\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d"
Sep 16 18:28:59.725046 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.724892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2588625-c9a6-40bf-a6b8-e55b26dfa2cf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7887965c76-tzl7q\" (UID: \"c2588625-c9a6-40bf-a6b8-e55b26dfa2cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q"
Sep 16 18:28:59.725046 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.724931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjvc\" (UniqueName: \"kubernetes.io/projected/c2588625-c9a6-40bf-a6b8-e55b26dfa2cf-kube-api-access-pdjvc\") pod \"managed-serviceaccount-addon-agent-7887965c76-tzl7q\" (UID: \"c2588625-c9a6-40bf-a6b8-e55b26dfa2cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q"
Sep 16 18:28:59.727269 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.727248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2588625-c9a6-40bf-a6b8-e55b26dfa2cf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7887965c76-tzl7q\" (UID: \"c2588625-c9a6-40bf-a6b8-e55b26dfa2cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q"
Sep 16 18:28:59.734725 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.734702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjvc\" (UniqueName: \"kubernetes.io/projected/c2588625-c9a6-40bf-a6b8-e55b26dfa2cf-kube-api-access-pdjvc\") pod \"managed-serviceaccount-addon-agent-7887965c76-tzl7q\" (UID: \"c2588625-c9a6-40bf-a6b8-e55b26dfa2cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q"
Sep 16 18:28:59.817827 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.817796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q"
Sep 16 18:28:59.825794 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.825770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b7e2fe2-787d-4a41-994b-d833daae8743-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2d65d\" (UID: \"9b7e2fe2-787d-4a41-994b-d833daae8743\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d"
Sep 16 18:28:59.825873 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.825804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b7e2fe2-787d-4a41-994b-d833daae8743-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2d65d\" (UID: \"9b7e2fe2-787d-4a41-994b-d833daae8743\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d"
Sep 16 18:28:59.826457 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.826435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b7e2fe2-787d-4a41-994b-d833daae8743-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2d65d\" (UID: \"9b7e2fe2-787d-4a41-994b-d833daae8743\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d"
Sep 16 18:28:59.828110 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.828085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b7e2fe2-787d-4a41-994b-d833daae8743-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2d65d\" (UID: \"9b7e2fe2-787d-4a41-994b-d833daae8743\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d"
Sep 16 18:28:59.902953 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:28:59.902921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d"
Sep 16 18:29:00.085451 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:00.085282    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q"]
Sep 16 18:29:00.089040 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:29:00.089010    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2588625_c9a6_40bf_a6b8_e55b26dfa2cf.slice/crio-0bc2c87c9f6da780af6aec40d03a70b56a20707f8f3c27a4f8bff648b9529963 WatchSource:0}: Error finding container 0bc2c87c9f6da780af6aec40d03a70b56a20707f8f3c27a4f8bff648b9529963: Status 404 returned error can't find the container with id 0bc2c87c9f6da780af6aec40d03a70b56a20707f8f3c27a4f8bff648b9529963
Sep 16 18:29:00.174843 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:00.174815    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d"]
Sep 16 18:29:00.177928 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:29:00.177899    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7e2fe2_787d_4a41_994b_d833daae8743.slice/crio-c5003c80fe3ccd0489a0e14d10097517ff674e64b2ed73fb3dae9a0b54e5180d WatchSource:0}: Error finding container c5003c80fe3ccd0489a0e14d10097517ff674e64b2ed73fb3dae9a0b54e5180d: Status 404 returned error can't find the container with id c5003c80fe3ccd0489a0e14d10097517ff674e64b2ed73fb3dae9a0b54e5180d
Sep 16 18:29:00.200594 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:00.200562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d" event={"ID":"9b7e2fe2-787d-4a41-994b-d833daae8743","Type":"ContainerStarted","Data":"c5003c80fe3ccd0489a0e14d10097517ff674e64b2ed73fb3dae9a0b54e5180d"}
Sep 16 18:29:00.201651 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:00.201631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q" event={"ID":"c2588625-c9a6-40bf-a6b8-e55b26dfa2cf","Type":"ContainerStarted","Data":"0bc2c87c9f6da780af6aec40d03a70b56a20707f8f3c27a4f8bff648b9529963"}
Sep 16 18:29:01.376774 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:01.376741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:29:01.377159 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:01.376800    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:29:01.377159 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:01.377130    2580 scope.go:117] "RemoveContainer" containerID="23f0c8ccfa4e99e0210ceed89f008e3203671bcabb5bb990370e3a990a415d4c"
Sep 16 18:29:01.377285 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:29:01.377267    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-bbkt2_openshift-console-operator(4222a382-cb85-4cd2-b8ed-0dd10fa26702)\"" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2" podUID="4222a382-cb85-4cd2-b8ed-0dd10fa26702"
Sep 16 18:29:02.210542 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:02.210496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d" event={"ID":"9b7e2fe2-787d-4a41-994b-d833daae8743","Type":"ContainerStarted","Data":"10ae6e236851e8c391e175e2d6524e550e74601fbb5da86e7bfff24e2e1ed509"}
Sep 16 18:29:02.239021 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:02.238963    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2d65d" podStartSLOduration=2.024973732 podStartE2EDuration="3.238940922s" totalImagesPullingTime="1.21396719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:02.237829532 +0000 UTC m=+84.947068923" watchObservedRunningTime="2026-09-16 18:29:02.238940922 +0000 UTC m=+84.948180313"
Sep 16 18:29:03.215093 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:03.215062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q" event={"ID":"c2588625-c9a6-40bf-a6b8-e55b26dfa2cf","Type":"ContainerStarted","Data":"5af4c702bd05b494f39f821f3af27ef2888f1ed70793d277bde7ff2a443477ad"}
Sep 16 18:29:03.233187 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:03.233142    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7887965c76-tzl7q" podStartSLOduration=1.201106349 podStartE2EDuration="4.233125589s" totalImagesPullingTime="3.03201924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:03.232217691 +0000 UTC m=+85.941457082" watchObservedRunningTime="2026-09-16 18:29:03.233125589 +0000 UTC m=+85.942364994"
Sep 16 18:29:06.771136 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:06.771104    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"]
Sep 16 18:29:06.774268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:06.774252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"
Sep 16 18:29:06.776604 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:06.776578    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:29:06.776729 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:06.776650    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nxd99\""
Sep 16 18:29:06.781629 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:06.781607    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"]
Sep 16 18:29:06.882978 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:06.882943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4bfef989-8ac2-465a-aac5-eabd93813538-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h2vzf\" (UID: \"4bfef989-8ac2-465a-aac5-eabd93813538\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"
Sep 16 18:29:06.983596 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:06.983563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4bfef989-8ac2-465a-aac5-eabd93813538-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h2vzf\" (UID: \"4bfef989-8ac2-465a-aac5-eabd93813538\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"
Sep 16 18:29:06.983715 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:29:06.983696    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:29:06.983777 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:29:06.983768    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bfef989-8ac2-465a-aac5-eabd93813538-tls-certificates podName:4bfef989-8ac2-465a-aac5-eabd93813538 nodeName:}" failed. No retries permitted until 2026-09-16 18:29:07.483752236 +0000 UTC m=+90.192991605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/4bfef989-8ac2-465a-aac5-eabd93813538-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-h2vzf" (UID: "4bfef989-8ac2-465a-aac5-eabd93813538") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:29:07.486912 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:07.486859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4bfef989-8ac2-465a-aac5-eabd93813538-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h2vzf\" (UID: \"4bfef989-8ac2-465a-aac5-eabd93813538\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"
Sep 16 18:29:07.487093 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:29:07.487018    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:29:07.487131 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:29:07.487098    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bfef989-8ac2-465a-aac5-eabd93813538-tls-certificates podName:4bfef989-8ac2-465a-aac5-eabd93813538 nodeName:}" failed. No retries permitted until 2026-09-16 18:29:08.487081601 +0000 UTC m=+91.196320971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/4bfef989-8ac2-465a-aac5-eabd93813538-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-h2vzf" (UID: "4bfef989-8ac2-465a-aac5-eabd93813538") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:29:08.492941 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:08.492912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4bfef989-8ac2-465a-aac5-eabd93813538-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h2vzf\" (UID: \"4bfef989-8ac2-465a-aac5-eabd93813538\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"
Sep 16 18:29:08.495360 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:08.495316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4bfef989-8ac2-465a-aac5-eabd93813538-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h2vzf\" (UID: \"4bfef989-8ac2-465a-aac5-eabd93813538\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"
Sep 16 18:29:08.583285 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:08.583262    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"
Sep 16 18:29:08.849932 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:08.849902    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"]
Sep 16 18:29:08.852531 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:29:08.852496    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfef989_8ac2_465a_aac5_eabd93813538.slice/crio-796f3bfbd49a766f2d6512d366c071612c9c590546c46c7ef6fb1b1fbb26e8b3 WatchSource:0}: Error finding container 796f3bfbd49a766f2d6512d366c071612c9c590546c46c7ef6fb1b1fbb26e8b3: Status 404 returned error can't find the container with id 796f3bfbd49a766f2d6512d366c071612c9c590546c46c7ef6fb1b1fbb26e8b3
Sep 16 18:29:09.234776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:09.234690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf" event={"ID":"4bfef989-8ac2-465a-aac5-eabd93813538","Type":"ContainerStarted","Data":"796f3bfbd49a766f2d6512d366c071612c9c590546c46c7ef6fb1b1fbb26e8b3"}
Sep 16 18:29:10.242857 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.242775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf" event={"ID":"4bfef989-8ac2-465a-aac5-eabd93813538","Type":"ContainerStarted","Data":"fcc13ca0abf70197f6cc88b4448c90227a58ce291e7b898a401f85723118c76a"}
Sep 16 18:29:10.243221 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.242993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"
Sep 16 18:29:10.247452 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.247434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf"
Sep 16 18:29:10.259608 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.259567    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h2vzf" podStartSLOduration=3.197513478 podStartE2EDuration="4.25955478s" totalImagesPullingTime="1.062041302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:10.259005713 +0000 UTC m=+92.968245104" watchObservedRunningTime="2026-09-16 18:29:10.25955478 +0000 UTC m=+92.968794173"
Sep 16 18:29:10.839241 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.839206    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"]
Sep 16 18:29:10.841993 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.841975    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:10.847536 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.847508    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:29:10.847657 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.847535    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:29:10.847657 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.847515    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:29:10.848634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.848615    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:29:10.848927 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.848911    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vb9cr\""
Sep 16 18:29:10.849024 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.848999    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:29:10.853672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.853652    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"]
Sep 16 18:29:10.911981 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.911944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8009d09-30df-4e68-9b01-a7b39cc41763-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:10.912141 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.911995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8009d09-30df-4e68-9b01-a7b39cc41763-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:10.912141 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.912126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wpw5\" (UniqueName: \"kubernetes.io/projected/c8009d09-30df-4e68-9b01-a7b39cc41763-kube-api-access-9wpw5\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:10.912226 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:10.912161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8009d09-30df-4e68-9b01-a7b39cc41763-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:11.013340 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:11.013304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wpw5\" (UniqueName: \"kubernetes.io/projected/c8009d09-30df-4e68-9b01-a7b39cc41763-kube-api-access-9wpw5\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:11.013492 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:11.013365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8009d09-30df-4e68-9b01-a7b39cc41763-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:11.013492 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:11.013396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8009d09-30df-4e68-9b01-a7b39cc41763-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:11.013492 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:11.013436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8009d09-30df-4e68-9b01-a7b39cc41763-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:11.013634 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:29:11.013537    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 18:29:11.013634 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:29:11.013615    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8009d09-30df-4e68-9b01-a7b39cc41763-prometheus-operator-tls podName:c8009d09-30df-4e68-9b01-a7b39cc41763 nodeName:}" failed. No retries permitted until 2026-09-16 18:29:11.513599518 +0000 UTC m=+94.222838889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c8009d09-30df-4e68-9b01-a7b39cc41763-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-6j8xp" (UID: "c8009d09-30df-4e68-9b01-a7b39cc41763") : secret "prometheus-operator-tls" not found
Sep 16 18:29:11.014080 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:11.014059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8009d09-30df-4e68-9b01-a7b39cc41763-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:11.016044 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:11.016018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8009d09-30df-4e68-9b01-a7b39cc41763-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:11.023493 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:11.023472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wpw5\" (UniqueName: \"kubernetes.io/projected/c8009d09-30df-4e68-9b01-a7b39cc41763-kube-api-access-9wpw5\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:11.516980 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:11.516930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8009d09-30df-4e68-9b01-a7b39cc41763-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:11.519343 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:11.519313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8009d09-30df-4e68-9b01-a7b39cc41763-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-6j8xp\" (UID: \"c8009d09-30df-4e68-9b01-a7b39cc41763\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:11.750671 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:11.750631    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"
Sep 16 18:29:12.020320 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:12.020295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp"]
Sep 16 18:29:12.022655 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:29:12.022627    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8009d09_30df_4e68_9b01_a7b39cc41763.slice/crio-a9ec2df96f46cea39ecc6d120e5d1e3fda947bf2e27ef8a575cab1b4f03477b5 WatchSource:0}: Error finding container a9ec2df96f46cea39ecc6d120e5d1e3fda947bf2e27ef8a575cab1b4f03477b5: Status 404 returned error can't find the container with id a9ec2df96f46cea39ecc6d120e5d1e3fda947bf2e27ef8a575cab1b4f03477b5
Sep 16 18:29:12.250301 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:12.250226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp" event={"ID":"c8009d09-30df-4e68-9b01-a7b39cc41763","Type":"ContainerStarted","Data":"a9ec2df96f46cea39ecc6d120e5d1e3fda947bf2e27ef8a575cab1b4f03477b5"}
Sep 16 18:29:13.255234 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:13.255207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp" event={"ID":"c8009d09-30df-4e68-9b01-a7b39cc41763","Type":"ContainerStarted","Data":"472b0d70d3fcce6601513798cd13ca3b41590464df50cb5de21a6b1b2fe20dad"}
Sep 16 18:29:14.260801 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:14.260764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp" event={"ID":"c8009d09-30df-4e68-9b01-a7b39cc41763","Type":"ContainerStarted","Data":"200bbc39f962d4656bf68ecc61787e500fe4221cf23da5639a229fa06f345ab9"}
Sep 16 18:29:14.277917 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:14.277865    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6j8xp" podStartSLOduration=3.128304604 podStartE2EDuration="4.277850065s" totalImagesPullingTime="1.149545461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:14.276408667 +0000 UTC m=+96.985648058" watchObservedRunningTime="2026-09-16 18:29:14.277850065 +0000 UTC m=+96.987089458"
Sep 16 18:29:14.777798 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:14.777765    2580 scope.go:117] "RemoveContainer" containerID="23f0c8ccfa4e99e0210ceed89f008e3203671bcabb5bb990370e3a990a415d4c"
Sep 16 18:29:15.046792 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.046705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:29:15.046792 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.046745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:29:15.049037 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.049014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef3fcd16-81ea-4a0a-8b03-f63208709c03-metrics-tls\") pod \"dns-default-8wtgj\" (UID: \"ef3fcd16-81ea-4a0a-8b03-f63208709c03\") " pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:29:15.049165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.049146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") pod \"image-registry-66cbbd67bc-tzfsk\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") " pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:29:15.267897 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.267869    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:29:15.268486 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.268459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2" event={"ID":"4222a382-cb85-4cd2-b8ed-0dd10fa26702","Type":"ContainerStarted","Data":"33a97ea5cd5403c296d246fc685cdcef6ba1efe4a4dbda69eb9fbdf28834d7c7"}
Sep 16 18:29:15.268856 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.268805    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:29:15.286999 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.286950    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2" podStartSLOduration=41.36662947 podStartE2EDuration="44.286939701s" totalImagesPullingTime="2.920310231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:15.286178259 +0000 UTC m=+97.995417650" watchObservedRunningTime="2026-09-16 18:29:15.286939701 +0000 UTC m=+97.996179126"
Sep 16 18:29:15.302854 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.302802    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jnwzm\""
Sep 16 18:29:15.310900 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.310886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:29:15.328193 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.328171    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sr6v8\""
Sep 16 18:29:15.336012 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.335990    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:29:15.467563 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.467531    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-bbkt2"
Sep 16 18:29:15.589599 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.589569    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66cbbd67bc-tzfsk"]
Sep 16 18:29:15.591062 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:29:15.591032    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bf4a5a_68bb_43ef_a991_57aee91b2fc4.slice/crio-2d3553acd3afa27e43e83adee68d7a6fa9b67ecebc61a5fec38a2e001b6c6f47 WatchSource:0}: Error finding container 2d3553acd3afa27e43e83adee68d7a6fa9b67ecebc61a5fec38a2e001b6c6f47: Status 404 returned error can't find the container with id 2d3553acd3afa27e43e83adee68d7a6fa9b67ecebc61a5fec38a2e001b6c6f47
Sep 16 18:29:15.626267 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:15.626238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8wtgj"]
Sep 16 18:29:15.629654 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:29:15.629609    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3fcd16_81ea_4a0a_8b03_f63208709c03.slice/crio-0c6a15f2c05318572935119682839534cb675beb289792b72e40de248314d82e WatchSource:0}: Error finding container 0c6a15f2c05318572935119682839534cb675beb289792b72e40de248314d82e: Status 404 returned error can't find the container with id 0c6a15f2c05318572935119682839534cb675beb289792b72e40de248314d82e
Sep 16 18:29:16.264145 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.264103    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m8qg9"]
Sep 16 18:29:16.267502 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.267482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.272126 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.272089    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:29:16.272523 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.272092    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-22lf8\""
Sep 16 18:29:16.272865 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.272715    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:29:16.272972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.272943    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:29:16.273484 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.273455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8wtgj" event={"ID":"ef3fcd16-81ea-4a0a-8b03-f63208709c03","Type":"ContainerStarted","Data":"0c6a15f2c05318572935119682839534cb675beb289792b72e40de248314d82e"}
Sep 16 18:29:16.275153 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.275126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk" event={"ID":"21bf4a5a-68bb-43ef-a991-57aee91b2fc4","Type":"ContainerStarted","Data":"9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96"}
Sep 16 18:29:16.275255 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.275176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk" event={"ID":"21bf4a5a-68bb-43ef-a991-57aee91b2fc4","Type":"ContainerStarted","Data":"2d3553acd3afa27e43e83adee68d7a6fa9b67ecebc61a5fec38a2e001b6c6f47"}
Sep 16 18:29:16.275255 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.275204    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:29:16.331198 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.331142    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk" podStartSLOduration=98.331121369 podStartE2EDuration="1m38.331121369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:16.329931112 +0000 UTC m=+99.039170507" watchObservedRunningTime="2026-09-16 18:29:16.331121369 +0000 UTC m=+99.040360772"
Sep 16 18:29:16.356438 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.356404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-wtmp\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.356438 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.356442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-root\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.356670 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.356467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.356670 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.356625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-sys\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.356776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.356704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.356921 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.356795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfbqw\" (UniqueName: \"kubernetes.io/projected/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-kube-api-access-pfbqw\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.356921 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.356825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-textfile\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.356921 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.356880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-tls\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.356921 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.356898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-metrics-client-ca\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.457993 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.457961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-wtmp\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.457993 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.457997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-root\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458191 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458191 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-root\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458265 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-sys\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458265 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-wtmp\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458265 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458265 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-sys\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458480 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfbqw\" (UniqueName: \"kubernetes.io/projected/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-kube-api-access-pfbqw\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458480 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-textfile\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458480 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-tls\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458480 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-metrics-client-ca\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458730 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-textfile\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458852 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.458912 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.458864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-metrics-client-ca\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.460363 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.460323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.460720 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.460698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-node-exporter-tls\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.467008 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.466986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfbqw\" (UniqueName: \"kubernetes.io/projected/016d2d6f-9c6f-4987-8d87-0aa89fdc568f-kube-api-access-pfbqw\") pod \"node-exporter-m8qg9\" (UID: \"016d2d6f-9c6f-4987-8d87-0aa89fdc568f\") " pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.577779 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:16.577746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m8qg9"
Sep 16 18:29:16.591275 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:29:16.591247    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016d2d6f_9c6f_4987_8d87_0aa89fdc568f.slice/crio-ef8827e3285e1c46a52659be86e6ce989a9f6000702fa190319102864d61ee31 WatchSource:0}: Error finding container ef8827e3285e1c46a52659be86e6ce989a9f6000702fa190319102864d61ee31: Status 404 returned error can't find the container with id ef8827e3285e1c46a52659be86e6ce989a9f6000702fa190319102864d61ee31
Sep 16 18:29:17.286616 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:17.286569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8qg9" event={"ID":"016d2d6f-9c6f-4987-8d87-0aa89fdc568f","Type":"ContainerStarted","Data":"ef8827e3285e1c46a52659be86e6ce989a9f6000702fa190319102864d61ee31"}
Sep 16 18:29:18.296143 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:18.296100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8wtgj" event={"ID":"ef3fcd16-81ea-4a0a-8b03-f63208709c03","Type":"ContainerStarted","Data":"cdab84730a21fff33697dd2ecec861031c247e32f4dbf45a293652a05a955eb2"}
Sep 16 18:29:18.296591 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:18.296162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8wtgj" event={"ID":"ef3fcd16-81ea-4a0a-8b03-f63208709c03","Type":"ContainerStarted","Data":"ab22811a517259588a53e70a6a49cdc4809061df09d27103b37bc3a2ca120195"}
Sep 16 18:29:18.296591 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:18.296218    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:29:18.297710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:18.297684    2580 generic.go:356] "Generic (PLEG): container finished" podID="016d2d6f-9c6f-4987-8d87-0aa89fdc568f" containerID="79f84c36bc0e555c5b3d504cfe3d2ee23853db314cf24d5a6f9a910c09eace13" exitCode=0
Sep 16 18:29:18.297805 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:18.297739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8qg9" event={"ID":"016d2d6f-9c6f-4987-8d87-0aa89fdc568f","Type":"ContainerDied","Data":"79f84c36bc0e555c5b3d504cfe3d2ee23853db314cf24d5a6f9a910c09eace13"}
Sep 16 18:29:18.314349 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:18.314288    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8wtgj" podStartSLOduration=65.142749731 podStartE2EDuration="1m7.314270727s" totalImagesPullingTime="2.171520996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:18.31319376 +0000 UTC m=+101.022433154" watchObservedRunningTime="2026-09-16 18:29:18.314270727 +0000 UTC m=+101.023510119"
Sep 16 18:29:19.303805 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:19.303763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8qg9" event={"ID":"016d2d6f-9c6f-4987-8d87-0aa89fdc568f","Type":"ContainerStarted","Data":"2724759f78099d4f5092ca561898f45e3e6d9ff8234e0c238c21d0da2dd4b6e1"}
Sep 16 18:29:19.304189 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:19.303830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8qg9" event={"ID":"016d2d6f-9c6f-4987-8d87-0aa89fdc568f","Type":"ContainerStarted","Data":"453670b0c3a32a9e709d3431391eda40de42413b7e7cceb9c9111984b2ac2a8f"}
Sep 16 18:29:19.330074 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:19.327444    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m8qg9" podStartSLOduration=2.043986401 podStartE2EDuration="3.327423368s" totalImagesPullingTime="1.209563532s" totalInitContainerRuntime="73.873435ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:19.326490031 +0000 UTC m=+102.035729426" watchObservedRunningTime="2026-09-16 18:29:19.327423368 +0000 UTC m=+102.036662764"
Sep 16 18:29:20.153384 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.153355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rxwgv"
Sep 16 18:29:20.642455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.642418    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-857fb664c9-fd5kt"]
Sep 16 18:29:20.645699 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.645683    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.648243 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.648225    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:29:20.649193 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.649171    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:29:20.649193 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.649183    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hcktf\""
Sep 16 18:29:20.649193 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.649194    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:29:20.649435 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.649210    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:29:20.649435 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.649211    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8so1j9686ov0b\""
Sep 16 18:29:20.655600 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.655580    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-857fb664c9-fd5kt"]
Sep 16 18:29:20.794800 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.794759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4acb3796-9cdf-408e-8b33-125d56846404-secret-metrics-server-client-certs\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.794800 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.794799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmm8q\" (UniqueName: \"kubernetes.io/projected/4acb3796-9cdf-408e-8b33-125d56846404-kube-api-access-kmm8q\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.795049 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.794890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4acb3796-9cdf-408e-8b33-125d56846404-audit-log\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.795049 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.794958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4acb3796-9cdf-408e-8b33-125d56846404-metrics-server-audit-profiles\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.795049 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.794981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4acb3796-9cdf-408e-8b33-125d56846404-client-ca-bundle\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.795049 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.795015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4acb3796-9cdf-408e-8b33-125d56846404-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.795049 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.795043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4acb3796-9cdf-408e-8b33-125d56846404-secret-metrics-server-tls\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.896308 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.896209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4acb3796-9cdf-408e-8b33-125d56846404-audit-log\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.896308 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.896264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4acb3796-9cdf-408e-8b33-125d56846404-metrics-server-audit-profiles\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.896308 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.896292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4acb3796-9cdf-408e-8b33-125d56846404-client-ca-bundle\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.896573 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.896322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4acb3796-9cdf-408e-8b33-125d56846404-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.896573 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.896385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4acb3796-9cdf-408e-8b33-125d56846404-secret-metrics-server-tls\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.896573 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.896423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4acb3796-9cdf-408e-8b33-125d56846404-secret-metrics-server-client-certs\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.896573 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.896453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmm8q\" (UniqueName: \"kubernetes.io/projected/4acb3796-9cdf-408e-8b33-125d56846404-kube-api-access-kmm8q\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.896856 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.896830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4acb3796-9cdf-408e-8b33-125d56846404-audit-log\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.897407 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.897369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4acb3796-9cdf-408e-8b33-125d56846404-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.897538 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.897522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4acb3796-9cdf-408e-8b33-125d56846404-metrics-server-audit-profiles\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.899477 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.899453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4acb3796-9cdf-408e-8b33-125d56846404-client-ca-bundle\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.899580 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.899493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4acb3796-9cdf-408e-8b33-125d56846404-secret-metrics-server-client-certs\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.899580 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.899532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4acb3796-9cdf-408e-8b33-125d56846404-secret-metrics-server-tls\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.904944 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.904924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmm8q\" (UniqueName: \"kubernetes.io/projected/4acb3796-9cdf-408e-8b33-125d56846404-kube-api-access-kmm8q\") pod \"metrics-server-857fb664c9-fd5kt\" (UID: \"4acb3796-9cdf-408e-8b33-125d56846404\") " pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:20.955383 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:20.955355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:21.230187 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:21.230001    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-857fb664c9-fd5kt"]
Sep 16 18:29:21.232751 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:29:21.232728    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acb3796_9cdf_408e_8b33_125d56846404.slice/crio-2fc461b2f18e8f61af7cd6b9a16b5e6a5e8e0a36125d7f0eeac42b0816871482 WatchSource:0}: Error finding container 2fc461b2f18e8f61af7cd6b9a16b5e6a5e8e0a36125d7f0eeac42b0816871482: Status 404 returned error can't find the container with id 2fc461b2f18e8f61af7cd6b9a16b5e6a5e8e0a36125d7f0eeac42b0816871482
Sep 16 18:29:21.311688 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:21.311655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt" event={"ID":"4acb3796-9cdf-408e-8b33-125d56846404","Type":"ContainerStarted","Data":"2fc461b2f18e8f61af7cd6b9a16b5e6a5e8e0a36125d7f0eeac42b0816871482"}
Sep 16 18:29:21.503887 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:21.503807    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66cbbd67bc-tzfsk"]
Sep 16 18:29:22.434100 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.434062    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:29:22.448626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.448598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.453581 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.453109    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:29:22.455339 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.454015    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:29:22.458445 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.458417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:29:22.460715 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.460687    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:29:22.461516 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.461496    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:29:22.461811 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.461691    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:29:22.462553 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.462533    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-edgic83qddbq1\""
Sep 16 18:29:22.463287 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.462874    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:29:22.464138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.463676    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:29:22.464266 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.463709    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:29:22.464394 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.464378    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:29:22.464874 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.463777    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:29:22.464874 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.463854    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9vddn\""
Sep 16 18:29:22.464874 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.463866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:29:22.464874 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.464407    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:29:22.464874 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.464448    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:29:22.465168 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.463768    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:29:22.611514 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-web-config\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611709 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611709 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611709 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611709 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-config-out\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611709 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-config\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611784    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9lmk\" (UniqueName: \"kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-kube-api-access-b9lmk\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.611974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.611944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.612314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.612009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.612314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.612036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.612314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.612077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.612314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.612098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.612314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.612124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.612314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.612140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.612314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.612167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.712688 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.712609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.712688 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.712653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.712688 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.712685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-config-out\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.712948 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.712711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713118 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-config\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9lmk\" (UniqueName: \"kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-kube-api-access-b9lmk\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713864 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713864 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713864 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-web-config\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.713864 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.713605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.715970 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.714756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.715970 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.715236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.715970 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.715314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.715970 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.715949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.717318 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.717009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.718005 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.717624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.718005 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.717940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.718920 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.718845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.719393 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.719246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-config-out\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.720261 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.719790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.720261 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.719841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.720261 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.720208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.721236 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.720805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.721236 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.721034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-config\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.721508 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.721473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.722133 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.722113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-web-config\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.723258 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.723214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.723960 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.723921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9lmk\" (UniqueName: \"kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-kube-api-access-b9lmk\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.724624 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.724598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.724847 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.724811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:22.772647 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:22.772422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:23.193339 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:23.193303    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:29:23.195053 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:29:23.195027    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65707337_adee_4567_b3e6_b7f38e576360.slice/crio-da8786e78d09b51471e8dd185ab2b9efafa4c98733996d1a256679f4b2834249 WatchSource:0}: Error finding container da8786e78d09b51471e8dd185ab2b9efafa4c98733996d1a256679f4b2834249: Status 404 returned error can't find the container with id da8786e78d09b51471e8dd185ab2b9efafa4c98733996d1a256679f4b2834249
Sep 16 18:29:23.320982 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:23.320953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt" event={"ID":"4acb3796-9cdf-408e-8b33-125d56846404","Type":"ContainerStarted","Data":"90564d01aedd84f6b31def9f38d7337dfc7b222cf3589a94d6e588a9ddde8229"}
Sep 16 18:29:23.322349 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:23.322295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerStarted","Data":"da8786e78d09b51471e8dd185ab2b9efafa4c98733996d1a256679f4b2834249"}
Sep 16 18:29:24.347534 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:24.347476    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt" podStartSLOduration=2.380585095 podStartE2EDuration="4.3474598s" totalImagesPullingTime="1.966874705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:24.346903169 +0000 UTC m=+107.056142574" watchObservedRunningTime="2026-09-16 18:29:24.3474598 +0000 UTC m=+107.056699284"
Sep 16 18:29:25.336070 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:25.336033    2580 generic.go:356] "Generic (PLEG): container finished" podID="65707337-adee-4567-b3e6-b7f38e576360" containerID="14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28" exitCode=0
Sep 16 18:29:25.336241 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:25.336105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerDied","Data":"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"}
Sep 16 18:29:28.306812 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:28.306782    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8wtgj"
Sep 16 18:29:29.356169 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:29.356124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerStarted","Data":"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"}
Sep 16 18:29:29.356596 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:29.356201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerStarted","Data":"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"}
Sep 16 18:29:31.369233 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:31.369198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerStarted","Data":"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"}
Sep 16 18:29:31.369641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:31.369271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerStarted","Data":"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"}
Sep 16 18:29:31.369641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:31.369301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerStarted","Data":"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"}
Sep 16 18:29:31.369641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:31.369350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerStarted","Data":"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"}
Sep 16 18:29:31.406694 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:31.406646    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.606193 podStartE2EDuration="9.406627653s" totalImagesPullingTime="6.776950919s" totalInitContainerRuntime="23.483734ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:29:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:31.403780221 +0000 UTC m=+114.113019614" watchObservedRunningTime="2026-09-16 18:29:31.406627653 +0000 UTC m=+114.115867044"
Sep 16 18:29:31.510574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:31.510546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:29:32.773689 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:32.773660    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:29:40.955574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:40.955538    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:40.956125 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:40.955609    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:29:46.423812 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.423781    2580 generic.go:356] "Generic (PLEG): container finished" podID="4d46e260-7277-4f07-8087-6b3d867252af" containerID="460c28fd41b79da803228b32037bd71dbd59c1d5cda62387a18ac65581224fc2" exitCode=0
Sep 16 18:29:46.424198 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.423855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24" event={"ID":"4d46e260-7277-4f07-8087-6b3d867252af","Type":"ContainerDied","Data":"460c28fd41b79da803228b32037bd71dbd59c1d5cda62387a18ac65581224fc2"}
Sep 16 18:29:46.424198 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.424175    2580 scope.go:117] "RemoveContainer" containerID="460c28fd41b79da803228b32037bd71dbd59c1d5cda62387a18ac65581224fc2"
Sep 16 18:29:46.522592 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.522553    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk" podUID="21bf4a5a-68bb-43ef-a991-57aee91b2fc4" containerName="registry" containerID="cri-o://9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96" gracePeriod=30
Sep 16 18:29:46.767815 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.767790    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:29:46.831458 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.831429    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-image-registry-private-configuration\") pod \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") "
Sep 16 18:29:46.831634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.831468    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkqsk\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-kube-api-access-bkqsk\") pod \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") "
Sep 16 18:29:46.831634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.831498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-ca-trust-extracted\") pod \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") "
Sep 16 18:29:46.831634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.831521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-trusted-ca\") pod \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") "
Sep 16 18:29:46.831795 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.831703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-installation-pull-secrets\") pod \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") "
Sep 16 18:29:46.831795 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.831748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-certificates\") pod \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") "
Sep 16 18:29:46.831795 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.831781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") pod \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") "
Sep 16 18:29:46.831949 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.831842    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-bound-sa-token\") pod \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\" (UID: \"21bf4a5a-68bb-43ef-a991-57aee91b2fc4\") "
Sep 16 18:29:46.832003 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.831965    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "21bf4a5a-68bb-43ef-a991-57aee91b2fc4" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:29:46.832310 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.832153    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-trusted-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:29:46.832310 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.832204    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "21bf4a5a-68bb-43ef-a991-57aee91b2fc4" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:29:46.834347 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.834307    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "21bf4a5a-68bb-43ef-a991-57aee91b2fc4" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:29:46.834489 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.834378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-kube-api-access-bkqsk" (OuterVolumeSpecName: "kube-api-access-bkqsk") pod "21bf4a5a-68bb-43ef-a991-57aee91b2fc4" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4"). InnerVolumeSpecName "kube-api-access-bkqsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:29:46.834545 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.834503    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "21bf4a5a-68bb-43ef-a991-57aee91b2fc4" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:29:46.834757 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.834728    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "21bf4a5a-68bb-43ef-a991-57aee91b2fc4" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:29:46.834833 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.834789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "21bf4a5a-68bb-43ef-a991-57aee91b2fc4" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:29:46.840486 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.840461    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "21bf4a5a-68bb-43ef-a991-57aee91b2fc4" (UID: "21bf4a5a-68bb-43ef-a991-57aee91b2fc4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:29:46.932831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.932803    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-image-registry-private-configuration\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:29:46.932831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.932826    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkqsk\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-kube-api-access-bkqsk\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:29:46.932974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.932836    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-ca-trust-extracted\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:29:46.932974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.932845    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-installation-pull-secrets\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:29:46.932974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.932854    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-certificates\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:29:46.932974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.932862    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-registry-tls\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:29:46.932974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:46.932871    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21bf4a5a-68bb-43ef-a991-57aee91b2fc4-bound-sa-token\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:29:47.430064 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.430028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-npd24" event={"ID":"4d46e260-7277-4f07-8087-6b3d867252af","Type":"ContainerStarted","Data":"101f0d964b56057d07b3dadde03d4800389579a72261e8471cd27c693c1697ba"}
Sep 16 18:29:47.431344 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.431303    2580 generic.go:356] "Generic (PLEG): container finished" podID="21bf4a5a-68bb-43ef-a991-57aee91b2fc4" containerID="9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96" exitCode=0
Sep 16 18:29:47.431531 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.431361    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk"
Sep 16 18:29:47.431531 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.431363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk" event={"ID":"21bf4a5a-68bb-43ef-a991-57aee91b2fc4","Type":"ContainerDied","Data":"9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96"}
Sep 16 18:29:47.431531 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.431406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66cbbd67bc-tzfsk" event={"ID":"21bf4a5a-68bb-43ef-a991-57aee91b2fc4","Type":"ContainerDied","Data":"2d3553acd3afa27e43e83adee68d7a6fa9b67ecebc61a5fec38a2e001b6c6f47"}
Sep 16 18:29:47.431531 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.431427    2580 scope.go:117] "RemoveContainer" containerID="9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96"
Sep 16 18:29:47.442670 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.442652    2580 scope.go:117] "RemoveContainer" containerID="9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96"
Sep 16 18:29:47.442946 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:29:47.442926    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96\": container with ID starting with 9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96 not found: ID does not exist" containerID="9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96"
Sep 16 18:29:47.442988 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.442956    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96"} err="failed to get container status \"9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96\": rpc error: code = NotFound desc = could not find container \"9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96\": container with ID starting with 9d3e79da8f7d8b757ebeae9203c3b0091632d7f496f359e111f3941c08035a96 not found: ID does not exist"
Sep 16 18:29:47.466456 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.466423    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66cbbd67bc-tzfsk"]
Sep 16 18:29:47.472884 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.472857    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66cbbd67bc-tzfsk"]
Sep 16 18:29:47.537434 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.537405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:29:47.539625 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.539604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e279fd-c891-476e-be69-1802a7262536-metrics-certs\") pod \"network-metrics-daemon-jd7wp\" (UID: \"58e279fd-c891-476e-be69-1802a7262536\") " pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:29:47.782022 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.781991    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21bf4a5a-68bb-43ef-a991-57aee91b2fc4" path="/var/lib/kubelet/pods/21bf4a5a-68bb-43ef-a991-57aee91b2fc4/volumes"
Sep 16 18:29:47.796686 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.796664    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p9wz2\""
Sep 16 18:29:47.804542 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:47.804524    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jd7wp"
Sep 16 18:29:48.080151 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:48.080077    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jd7wp"]
Sep 16 18:29:48.083185 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:29:48.083159    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e279fd_c891_476e_be69_1802a7262536.slice/crio-4070a0e37bbe56e70cc16a317692ae78e98889b9fe200bd5c828c6e710518af8 WatchSource:0}: Error finding container 4070a0e37bbe56e70cc16a317692ae78e98889b9fe200bd5c828c6e710518af8: Status 404 returned error can't find the container with id 4070a0e37bbe56e70cc16a317692ae78e98889b9fe200bd5c828c6e710518af8
Sep 16 18:29:48.436453 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:48.436365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jd7wp" event={"ID":"58e279fd-c891-476e-be69-1802a7262536","Type":"ContainerStarted","Data":"4070a0e37bbe56e70cc16a317692ae78e98889b9fe200bd5c828c6e710518af8"}
Sep 16 18:29:49.442670 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:49.442596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jd7wp" event={"ID":"58e279fd-c891-476e-be69-1802a7262536","Type":"ContainerStarted","Data":"b76b6a5316b3429e8da86b4c46a1cb8b284a59887d3d10be130e54f897275b05"}
Sep 16 18:29:49.442670 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:49.442649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jd7wp" event={"ID":"58e279fd-c891-476e-be69-1802a7262536","Type":"ContainerStarted","Data":"088172c501e6e39c1b8a542b602d1ed76605718e72fd6875f1b1921ebc0008e9"}
Sep 16 18:29:49.464276 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:29:49.464217    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jd7wp" podStartSLOduration=130.462053761 podStartE2EDuration="2m11.464199013s" totalImagesPullingTime="1.002145252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:49.464042422 +0000 UTC m=+132.173281817" watchObservedRunningTime="2026-09-16 18:29:49.464199013 +0000 UTC m=+132.173438405"
Sep 16 18:30:00.961254 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:00.961222    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:30:00.965180 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:00.965150    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-857fb664c9-fd5kt"
Sep 16 18:30:22.773632 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:22.773599    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:22.792221 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:22.792196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:23.590915 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:23.590888    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.086718 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.086679    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:30:41.087135 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.086985    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="prometheus" containerID="cri-o://67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b" gracePeriod=600
Sep 16 18:30:41.087135 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.087032    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="thanos-sidecar" containerID="cri-o://41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7" gracePeriod=600
Sep 16 18:30:41.087135 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.087053    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy" containerID="cri-o://729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561" gracePeriod=600
Sep 16 18:30:41.087135 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.087091    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy-web" containerID="cri-o://41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820" gracePeriod=600
Sep 16 18:30:41.087135 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.087112    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1" gracePeriod=600
Sep 16 18:30:41.087417 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.087060    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="config-reloader" containerID="cri-o://2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8" gracePeriod=600
Sep 16 18:30:41.328614 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.328583    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.371489 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371404    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-trusted-ca-bundle\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.371489 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-1\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.371489 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371481    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-db\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.371738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-tls-assets\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.371738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-kubelet-serving-ca-bundle\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.371738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371568    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-metrics-client-ca\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.371738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371593    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-2\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.371738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371637    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-web-config\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.371738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-serving-certs-ca-bundle\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.371738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-tls\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.371738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371723    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-config-out\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.372120 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371755    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-thanos-prometheus-http-client-file\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.372120 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371786    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:30:41.372120 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371804    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.372120 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-kube-rbac-proxy\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.372120 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371870    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-config\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.372120 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371919    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9lmk\" (UniqueName: \"kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-kube-api-access-b9lmk\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.372120 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371945    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-0\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.372120 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.371999    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-grpc-tls\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.372120 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.372024    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.372120 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.372060    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-metrics-client-certs\") pod \"65707337-adee-4567-b3e6-b7f38e576360\" (UID: \"65707337-adee-4567-b3e6-b7f38e576360\") "
Sep 16 18:30:41.372626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.372184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:30:41.372626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.372313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:30:41.372626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.372353    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.372626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.372377    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.372626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.372467    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:30:41.372884 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.372732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:30:41.373323 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.372949    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:30:41.373956 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.373924    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:30:41.375647 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.375597    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:30:41.376365 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.375826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-config-out" (OuterVolumeSpecName: "config-out") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:30:41.376365 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.375897    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:30:41.376365 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.375972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:30:41.376365 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.376013    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:30:41.377262 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.377235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:30:41.377778 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.377731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:30:41.377778 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.377749    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:30:41.377778 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.377748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-kube-api-access-b9lmk" (OuterVolumeSpecName: "kube-api-access-b9lmk") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "kube-api-access-b9lmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:30:41.377982 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.377873    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:30:41.378266 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.378230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:30:41.378918 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.378895    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-config" (OuterVolumeSpecName: "config") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:30:41.389516 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.389486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-web-config" (OuterVolumeSpecName: "web-config") pod "65707337-adee-4567-b3e6-b7f38e576360" (UID: "65707337-adee-4567-b3e6-b7f38e576360"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:30:41.473743 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473710    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-web-config\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473743 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473738    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473743 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473750    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473760    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-config-out\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473769    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473777    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473786    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-kube-rbac-proxy\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473796    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-config\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473804    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9lmk\" (UniqueName: \"kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-kube-api-access-b9lmk\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473812    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473820    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-grpc-tls\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473828    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473836    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65707337-adee-4567-b3e6-b7f38e576360-secret-metrics-client-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473843    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-db\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473853    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65707337-adee-4567-b3e6-b7f38e576360-tls-assets\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473861    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473868    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-configmap-metrics-client-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.473957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.473877    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65707337-adee-4567-b3e6-b7f38e576360-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:30:41.642649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642563    2580 generic.go:356] "Generic (PLEG): container finished" podID="65707337-adee-4567-b3e6-b7f38e576360" containerID="5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1" exitCode=0
Sep 16 18:30:41.642649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642590    2580 generic.go:356] "Generic (PLEG): container finished" podID="65707337-adee-4567-b3e6-b7f38e576360" containerID="729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561" exitCode=0
Sep 16 18:30:41.642649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642598    2580 generic.go:356] "Generic (PLEG): container finished" podID="65707337-adee-4567-b3e6-b7f38e576360" containerID="41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820" exitCode=0
Sep 16 18:30:41.642649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642603    2580 generic.go:356] "Generic (PLEG): container finished" podID="65707337-adee-4567-b3e6-b7f38e576360" containerID="41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7" exitCode=0
Sep 16 18:30:41.642649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642608    2580 generic.go:356] "Generic (PLEG): container finished" podID="65707337-adee-4567-b3e6-b7f38e576360" containerID="2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8" exitCode=0
Sep 16 18:30:41.642649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642614    2580 generic.go:356] "Generic (PLEG): container finished" podID="65707337-adee-4567-b3e6-b7f38e576360" containerID="67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b" exitCode=0
Sep 16 18:30:41.642649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642642    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.642972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerDied","Data":"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"}
Sep 16 18:30:41.642972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642718    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerDied","Data":"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"}
Sep 16 18:30:41.642972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerDied","Data":"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"}
Sep 16 18:30:41.642972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerDied","Data":"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"}
Sep 16 18:30:41.642972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642783    2580 scope.go:117] "RemoveContainer" containerID="5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"
Sep 16 18:30:41.642972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerDied","Data":"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"}
Sep 16 18:30:41.642972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerDied","Data":"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"}
Sep 16 18:30:41.642972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.642852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65707337-adee-4567-b3e6-b7f38e576360","Type":"ContainerDied","Data":"da8786e78d09b51471e8dd185ab2b9efafa4c98733996d1a256679f4b2834249"}
Sep 16 18:30:41.650998 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.650970    2580 scope.go:117] "RemoveContainer" containerID="729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"
Sep 16 18:30:41.658676 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.658656    2580 scope.go:117] "RemoveContainer" containerID="41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"
Sep 16 18:30:41.665848 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.665829    2580 scope.go:117] "RemoveContainer" containerID="41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"
Sep 16 18:30:41.670108 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.669982    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:30:41.672972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.672954    2580 scope.go:117] "RemoveContainer" containerID="2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"
Sep 16 18:30:41.679800 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.679777    2580 scope.go:117] "RemoveContainer" containerID="67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"
Sep 16 18:30:41.682003 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.681980    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:30:41.686757 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.686740    2580 scope.go:117] "RemoveContainer" containerID="14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"
Sep 16 18:30:41.692948 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.692917    2580 scope.go:117] "RemoveContainer" containerID="5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"
Sep 16 18:30:41.693198 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:30:41.693178    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": container with ID starting with 5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1 not found: ID does not exist" containerID="5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"
Sep 16 18:30:41.693244 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.693209    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"} err="failed to get container status \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": rpc error: code = NotFound desc = could not find container \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": container with ID starting with 5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1 not found: ID does not exist"
Sep 16 18:30:41.693244 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.693228    2580 scope.go:117] "RemoveContainer" containerID="729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"
Sep 16 18:30:41.693491 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:30:41.693476    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": container with ID starting with 729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561 not found: ID does not exist" containerID="729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"
Sep 16 18:30:41.693539 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.693493    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"} err="failed to get container status \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": rpc error: code = NotFound desc = could not find container \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": container with ID starting with 729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561 not found: ID does not exist"
Sep 16 18:30:41.693539 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.693506    2580 scope.go:117] "RemoveContainer" containerID="41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"
Sep 16 18:30:41.693768 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:30:41.693750    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": container with ID starting with 41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820 not found: ID does not exist" containerID="41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"
Sep 16 18:30:41.693808 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.693773    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"} err="failed to get container status \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": rpc error: code = NotFound desc = could not find container \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": container with ID starting with 41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820 not found: ID does not exist"
Sep 16 18:30:41.693808 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.693789    2580 scope.go:117] "RemoveContainer" containerID="41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"
Sep 16 18:30:41.694033 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:30:41.694016    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": container with ID starting with 41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7 not found: ID does not exist" containerID="41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"
Sep 16 18:30:41.694081 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.694038    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"} err="failed to get container status \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": rpc error: code = NotFound desc = could not find container \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": container with ID starting with 41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7 not found: ID does not exist"
Sep 16 18:30:41.694081 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.694053    2580 scope.go:117] "RemoveContainer" containerID="2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"
Sep 16 18:30:41.694292 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:30:41.694278    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": container with ID starting with 2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8 not found: ID does not exist" containerID="2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"
Sep 16 18:30:41.694355 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.694295    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"} err="failed to get container status \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": rpc error: code = NotFound desc = could not find container \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": container with ID starting with 2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8 not found: ID does not exist"
Sep 16 18:30:41.694355 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.694309    2580 scope.go:117] "RemoveContainer" containerID="67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"
Sep 16 18:30:41.694549 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:30:41.694529    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": container with ID starting with 67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b not found: ID does not exist" containerID="67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"
Sep 16 18:30:41.694603 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.694555    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"} err="failed to get container status \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": rpc error: code = NotFound desc = could not find container \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": container with ID starting with 67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b not found: ID does not exist"
Sep 16 18:30:41.694603 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.694569    2580 scope.go:117] "RemoveContainer" containerID="14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"
Sep 16 18:30:41.694794 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:30:41.694777    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": container with ID starting with 14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28 not found: ID does not exist" containerID="14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"
Sep 16 18:30:41.694833 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.694797    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"} err="failed to get container status \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": rpc error: code = NotFound desc = could not find container \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": container with ID starting with 14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28 not found: ID does not exist"
Sep 16 18:30:41.694833 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.694811    2580 scope.go:117] "RemoveContainer" containerID="5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"
Sep 16 18:30:41.695021 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.695003    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"} err="failed to get container status \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": rpc error: code = NotFound desc = could not find container \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": container with ID starting with 5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1 not found: ID does not exist"
Sep 16 18:30:41.695068 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.695022    2580 scope.go:117] "RemoveContainer" containerID="729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"
Sep 16 18:30:41.695228 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.695213    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"} err="failed to get container status \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": rpc error: code = NotFound desc = could not find container \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": container with ID starting with 729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561 not found: ID does not exist"
Sep 16 18:30:41.695289 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.695231    2580 scope.go:117] "RemoveContainer" containerID="41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"
Sep 16 18:30:41.695495 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.695478    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"} err="failed to get container status \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": rpc error: code = NotFound desc = could not find container \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": container with ID starting with 41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820 not found: ID does not exist"
Sep 16 18:30:41.695553 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.695496    2580 scope.go:117] "RemoveContainer" containerID="41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"
Sep 16 18:30:41.695737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.695717    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"} err="failed to get container status \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": rpc error: code = NotFound desc = could not find container \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": container with ID starting with 41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7 not found: ID does not exist"
Sep 16 18:30:41.695808 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.695740    2580 scope.go:117] "RemoveContainer" containerID="2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"
Sep 16 18:30:41.695976 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.695953    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"} err="failed to get container status \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": rpc error: code = NotFound desc = could not find container \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": container with ID starting with 2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8 not found: ID does not exist"
Sep 16 18:30:41.696019 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.695977    2580 scope.go:117] "RemoveContainer" containerID="67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"
Sep 16 18:30:41.696195 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.696178    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"} err="failed to get container status \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": rpc error: code = NotFound desc = could not find container \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": container with ID starting with 67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b not found: ID does not exist"
Sep 16 18:30:41.696235 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.696195    2580 scope.go:117] "RemoveContainer" containerID="14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"
Sep 16 18:30:41.696409 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.696395    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"} err="failed to get container status \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": rpc error: code = NotFound desc = could not find container \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": container with ID starting with 14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28 not found: ID does not exist"
Sep 16 18:30:41.696448 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.696409    2580 scope.go:117] "RemoveContainer" containerID="5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"
Sep 16 18:30:41.696610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.696593    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"} err="failed to get container status \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": rpc error: code = NotFound desc = could not find container \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": container with ID starting with 5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1 not found: ID does not exist"
Sep 16 18:30:41.696676 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.696612    2580 scope.go:117] "RemoveContainer" containerID="729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"
Sep 16 18:30:41.696830 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.696810    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"} err="failed to get container status \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": rpc error: code = NotFound desc = could not find container \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": container with ID starting with 729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561 not found: ID does not exist"
Sep 16 18:30:41.696878 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.696831    2580 scope.go:117] "RemoveContainer" containerID="41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"
Sep 16 18:30:41.697012 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.696998    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"} err="failed to get container status \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": rpc error: code = NotFound desc = could not find container \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": container with ID starting with 41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820 not found: ID does not exist"
Sep 16 18:30:41.697012 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.697011    2580 scope.go:117] "RemoveContainer" containerID="41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"
Sep 16 18:30:41.697186 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.697171    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"} err="failed to get container status \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": rpc error: code = NotFound desc = could not find container \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": container with ID starting with 41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7 not found: ID does not exist"
Sep 16 18:30:41.697226 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.697185    2580 scope.go:117] "RemoveContainer" containerID="2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"
Sep 16 18:30:41.697395 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.697378    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"} err="failed to get container status \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": rpc error: code = NotFound desc = could not find container \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": container with ID starting with 2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8 not found: ID does not exist"
Sep 16 18:30:41.697467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.697396    2580 scope.go:117] "RemoveContainer" containerID="67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"
Sep 16 18:30:41.697616 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.697601    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"} err="failed to get container status \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": rpc error: code = NotFound desc = could not find container \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": container with ID starting with 67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b not found: ID does not exist"
Sep 16 18:30:41.697663 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.697616    2580 scope.go:117] "RemoveContainer" containerID="14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"
Sep 16 18:30:41.697806 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.697789    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"} err="failed to get container status \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": rpc error: code = NotFound desc = could not find container \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": container with ID starting with 14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28 not found: ID does not exist"
Sep 16 18:30:41.697865 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.697808    2580 scope.go:117] "RemoveContainer" containerID="5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"
Sep 16 18:30:41.698028 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.698014    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"} err="failed to get container status \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": rpc error: code = NotFound desc = could not find container \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": container with ID starting with 5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1 not found: ID does not exist"
Sep 16 18:30:41.698071 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.698028    2580 scope.go:117] "RemoveContainer" containerID="729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"
Sep 16 18:30:41.698214 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.698200    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"} err="failed to get container status \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": rpc error: code = NotFound desc = could not find container \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": container with ID starting with 729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561 not found: ID does not exist"
Sep 16 18:30:41.698258 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.698215    2580 scope.go:117] "RemoveContainer" containerID="41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"
Sep 16 18:30:41.698436 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.698417    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"} err="failed to get container status \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": rpc error: code = NotFound desc = could not find container \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": container with ID starting with 41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820 not found: ID does not exist"
Sep 16 18:30:41.698509 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.698438    2580 scope.go:117] "RemoveContainer" containerID="41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"
Sep 16 18:30:41.698661 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.698647    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"} err="failed to get container status \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": rpc error: code = NotFound desc = could not find container \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": container with ID starting with 41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7 not found: ID does not exist"
Sep 16 18:30:41.698706 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.698661    2580 scope.go:117] "RemoveContainer" containerID="2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"
Sep 16 18:30:41.698860 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.698843    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"} err="failed to get container status \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": rpc error: code = NotFound desc = could not find container \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": container with ID starting with 2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8 not found: ID does not exist"
Sep 16 18:30:41.698925 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.698861    2580 scope.go:117] "RemoveContainer" containerID="67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"
Sep 16 18:30:41.699074 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.699057    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"} err="failed to get container status \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": rpc error: code = NotFound desc = could not find container \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": container with ID starting with 67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b not found: ID does not exist"
Sep 16 18:30:41.699119 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.699075    2580 scope.go:117] "RemoveContainer" containerID="14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"
Sep 16 18:30:41.699286 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.699271    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"} err="failed to get container status \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": rpc error: code = NotFound desc = could not find container \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": container with ID starting with 14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28 not found: ID does not exist"
Sep 16 18:30:41.699286 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.699285    2580 scope.go:117] "RemoveContainer" containerID="5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"
Sep 16 18:30:41.699493 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.699477    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"} err="failed to get container status \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": rpc error: code = NotFound desc = could not find container \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": container with ID starting with 5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1 not found: ID does not exist"
Sep 16 18:30:41.699558 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.699494    2580 scope.go:117] "RemoveContainer" containerID="729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"
Sep 16 18:30:41.699714 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.699699    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"} err="failed to get container status \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": rpc error: code = NotFound desc = could not find container \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": container with ID starting with 729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561 not found: ID does not exist"
Sep 16 18:30:41.699758 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.699714    2580 scope.go:117] "RemoveContainer" containerID="41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"
Sep 16 18:30:41.699899 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.699882    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"} err="failed to get container status \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": rpc error: code = NotFound desc = could not find container \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": container with ID starting with 41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820 not found: ID does not exist"
Sep 16 18:30:41.699969 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.699900    2580 scope.go:117] "RemoveContainer" containerID="41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"
Sep 16 18:30:41.700098 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.700083    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"} err="failed to get container status \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": rpc error: code = NotFound desc = could not find container \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": container with ID starting with 41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7 not found: ID does not exist"
Sep 16 18:30:41.700142 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.700098    2580 scope.go:117] "RemoveContainer" containerID="2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"
Sep 16 18:30:41.700277 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.700259    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"} err="failed to get container status \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": rpc error: code = NotFound desc = could not find container \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": container with ID starting with 2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8 not found: ID does not exist"
Sep 16 18:30:41.700315 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.700277    2580 scope.go:117] "RemoveContainer" containerID="67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"
Sep 16 18:30:41.700488 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.700464    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"} err="failed to get container status \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": rpc error: code = NotFound desc = could not find container \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": container with ID starting with 67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b not found: ID does not exist"
Sep 16 18:30:41.700539 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.700487    2580 scope.go:117] "RemoveContainer" containerID="14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"
Sep 16 18:30:41.700673 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.700653    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"} err="failed to get container status \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": rpc error: code = NotFound desc = could not find container \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": container with ID starting with 14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28 not found: ID does not exist"
Sep 16 18:30:41.700782 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.700674    2580 scope.go:117] "RemoveContainer" containerID="5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"
Sep 16 18:30:41.700866 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.700851    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1"} err="failed to get container status \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": rpc error: code = NotFound desc = could not find container \"5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1\": container with ID starting with 5e3f168f4411839177f7a3efae2ddb958b87401a9fd200c8a82d731fcdfba0e1 not found: ID does not exist"
Sep 16 18:30:41.700913 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.700866    2580 scope.go:117] "RemoveContainer" containerID="729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"
Sep 16 18:30:41.701042 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.701027    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561"} err="failed to get container status \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": rpc error: code = NotFound desc = could not find container \"729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561\": container with ID starting with 729d33721c9f1d7588c4534515e7218056c98397be7ba316feb6fc9b4eb93561 not found: ID does not exist"
Sep 16 18:30:41.701093 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.701042    2580 scope.go:117] "RemoveContainer" containerID="41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"
Sep 16 18:30:41.701204 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.701189    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820"} err="failed to get container status \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": rpc error: code = NotFound desc = could not find container \"41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820\": container with ID starting with 41d5b0dfa1eb5188b74acc4b7a1de02bfed8f55059e8c6a2e3812b467cf96820 not found: ID does not exist"
Sep 16 18:30:41.701204 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.701203    2580 scope.go:117] "RemoveContainer" containerID="41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"
Sep 16 18:30:41.701398 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.701375    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7"} err="failed to get container status \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": rpc error: code = NotFound desc = could not find container \"41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7\": container with ID starting with 41d8c1212c2e98899b3e75705c597812c86fbf07dc51023a3f0b373977aa16f7 not found: ID does not exist"
Sep 16 18:30:41.701398 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.701390    2580 scope.go:117] "RemoveContainer" containerID="2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"
Sep 16 18:30:41.701628 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.701611    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8"} err="failed to get container status \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": rpc error: code = NotFound desc = could not find container \"2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8\": container with ID starting with 2416b1e70e56c0bdc1e03348e740c08bf7072d9ce3b62b4034926c6afc07a3b8 not found: ID does not exist"
Sep 16 18:30:41.701685 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.701628    2580 scope.go:117] "RemoveContainer" containerID="67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"
Sep 16 18:30:41.701860 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.701840    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b"} err="failed to get container status \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": rpc error: code = NotFound desc = could not find container \"67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b\": container with ID starting with 67d7b30d61f4e11192025461d21aa207d9ba04a75b49021ed33d4e62786f390b not found: ID does not exist"
Sep 16 18:30:41.701933 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.701862    2580 scope.go:117] "RemoveContainer" containerID="14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"
Sep 16 18:30:41.702093 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.702078    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28"} err="failed to get container status \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": rpc error: code = NotFound desc = could not find container \"14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28\": container with ID starting with 14a0a60a32abf44de2b66a74a813d7f54049d3464d89659a8c51de1e99e93c28 not found: ID does not exist"
Sep 16 18:30:41.716286 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716259    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:30:41.716562 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716546    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="init-config-reloader"
Sep 16 18:30:41.716634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716565    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="init-config-reloader"
Sep 16 18:30:41.716634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716581    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="prometheus"
Sep 16 18:30:41.716634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716590    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="prometheus"
Sep 16 18:30:41.716634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716602    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy-thanos"
Sep 16 18:30:41.716634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716632    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy-thanos"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716646    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21bf4a5a-68bb-43ef-a991-57aee91b2fc4" containerName="registry"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716655    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="21bf4a5a-68bb-43ef-a991-57aee91b2fc4" containerName="registry"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716677    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="thanos-sidecar"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716685    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="thanos-sidecar"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716702    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="config-reloader"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716710    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="config-reloader"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716720    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy-web"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716728    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy-web"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716738    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716747    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716809    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy-web"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716820    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="prometheus"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716831    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="config-reloader"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716840    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="21bf4a5a-68bb-43ef-a991-57aee91b2fc4" containerName="registry"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716850    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716858    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="kube-rbac-proxy-thanos"
Sep 16 18:30:41.716861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.716868    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65707337-adee-4567-b3e6-b7f38e576360" containerName="thanos-sidecar"
Sep 16 18:30:41.722378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.722361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.724597 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.724582    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:30:41.724864 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.724845    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:30:41.724945 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.724889    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:30:41.724945 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.724902    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:30:41.724945 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.724851    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:30:41.725135 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.725122    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:30:41.725425 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.725406    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:30:41.725527 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.725504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:30:41.725618 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.725582    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9vddn\""
Sep 16 18:30:41.725761 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.725745    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:30:41.726088 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.726069    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:30:41.726216 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.726196    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:30:41.726299 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.726265    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-edgic83qddbq1\""
Sep 16 18:30:41.726961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.726940    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:30:41.728261 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.728064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:30:41.733081 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.733062    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:30:41.741788 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.741767    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:30:41.775553 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.775738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.775738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.775738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.775738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-config\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.775738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/367c7793-f1db-4a24-a898-ce6a170d79dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.775934 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.775934 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.775934 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.775934 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.775934 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lsjq\" (UniqueName: \"kubernetes.io/projected/367c7793-f1db-4a24-a898-ce6a170d79dc-kube-api-access-4lsjq\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.775934 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.775906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.776192 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.776045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.776192 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.776082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.776192 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.776151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.776192 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.776182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/367c7793-f1db-4a24-a898-ce6a170d79dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.776357 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.776205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.776357 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.776228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.776357 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.776247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.776357 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.776264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.781054 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.781029    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65707337-adee-4567-b3e6-b7f38e576360" path="/var/lib/kubelet/pods/65707337-adee-4567-b3e6-b7f38e576360/volumes"
Sep 16 18:30:41.876911 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.876871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.876911 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.876913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877157 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.876945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877157 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.876964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-config\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877157 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.876990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/367c7793-f1db-4a24-a898-ce6a170d79dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877295 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877295 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877295 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877295 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lsjq\" (UniqueName: \"kubernetes.io/projected/367c7793-f1db-4a24-a898-ce6a170d79dc-kube-api-access-4lsjq\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/367c7793-f1db-4a24-a898-ce6a170d79dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877914 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877914 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877914 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877914 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.877914 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.877784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.878679 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.878374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.878822 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.878796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.880085 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.880057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.880720 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.880693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.882348 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.881256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/367c7793-f1db-4a24-a898-ce6a170d79dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.882348 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.881518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.882348 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.881981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.883256 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.882774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.883256 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.882782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/367c7793-f1db-4a24-a898-ce6a170d79dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.883256 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.883063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.883256 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.883158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.883256 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.883241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-config\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.883584 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.883302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/367c7793-f1db-4a24-a898-ce6a170d79dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.883648 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.883626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.883802 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.883779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.884308 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.884286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.884840 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.884781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/367c7793-f1db-4a24-a898-ce6a170d79dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:41.902057 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:41.902009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lsjq\" (UniqueName: \"kubernetes.io/projected/367c7793-f1db-4a24-a898-ce6a170d79dc-kube-api-access-4lsjq\") pod \"prometheus-k8s-0\" (UID: \"367c7793-f1db-4a24-a898-ce6a170d79dc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:42.035199 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:42.035157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:30:42.333363 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:42.333321    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:30:42.334823 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:30:42.334797    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367c7793_f1db_4a24_a898_ce6a170d79dc.slice/crio-c3edcd0e64d00658d450253ae04749bfbd69523d91c8de38641dcb6e9390b71b WatchSource:0}: Error finding container c3edcd0e64d00658d450253ae04749bfbd69523d91c8de38641dcb6e9390b71b: Status 404 returned error can't find the container with id c3edcd0e64d00658d450253ae04749bfbd69523d91c8de38641dcb6e9390b71b
Sep 16 18:30:42.648194 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:42.648107    2580 generic.go:356] "Generic (PLEG): container finished" podID="367c7793-f1db-4a24-a898-ce6a170d79dc" containerID="b23d47245bdb2101d9563bfcb84b86f334dda58739ad620992be0db4c026a069" exitCode=0
Sep 16 18:30:42.648320 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:42.648195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"367c7793-f1db-4a24-a898-ce6a170d79dc","Type":"ContainerDied","Data":"b23d47245bdb2101d9563bfcb84b86f334dda58739ad620992be0db4c026a069"}
Sep 16 18:30:42.648320 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:42.648262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"367c7793-f1db-4a24-a898-ce6a170d79dc","Type":"ContainerStarted","Data":"c3edcd0e64d00658d450253ae04749bfbd69523d91c8de38641dcb6e9390b71b"}
Sep 16 18:30:43.655401 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:43.655363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"367c7793-f1db-4a24-a898-ce6a170d79dc","Type":"ContainerStarted","Data":"d24aae183829066307aea0e2f8f2ce7910c64761c37d6dac9f17148fbcd967c0"}
Sep 16 18:30:43.655763 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:43.655435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"367c7793-f1db-4a24-a898-ce6a170d79dc","Type":"ContainerStarted","Data":"86f77fefdd3a1d87e1dad3e5b559021f7cec6138887eeada0da123103a63ddfc"}
Sep 16 18:30:43.655763 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:43.655470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"367c7793-f1db-4a24-a898-ce6a170d79dc","Type":"ContainerStarted","Data":"4cf4d217aa0cb405942b78320af6e77fb79ae209307ccad2dcb5be07453ec9f3"}
Sep 16 18:30:43.655763 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:43.655495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"367c7793-f1db-4a24-a898-ce6a170d79dc","Type":"ContainerStarted","Data":"ffc22f34ce89f056ebf9a932e88130b0db739344ad78f26e8d8eea90a52f61ea"}
Sep 16 18:30:43.655763 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:43.655519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"367c7793-f1db-4a24-a898-ce6a170d79dc","Type":"ContainerStarted","Data":"ac9cb8982823bd514d104a0f5839bb52591e8d52727ad8d7c29da64bbb6cc69c"}
Sep 16 18:30:43.655763 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:43.655551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"367c7793-f1db-4a24-a898-ce6a170d79dc","Type":"ContainerStarted","Data":"38071ff0fbac2c4018744f09b7740c5ba5913eed9e7287b49b479b21b74604e6"}
Sep 16 18:30:43.692903 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:43.692853    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.673916013 podStartE2EDuration="2.692836695s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.920682ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:43.691273115 +0000 UTC m=+186.400512506" watchObservedRunningTime="2026-09-16 18:30:43.692836695 +0000 UTC m=+186.402076087"
Sep 16 18:30:47.035668 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:30:47.035617    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:31:42.035505 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:31:42.035467    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:31:42.050603 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:31:42.050582    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:31:42.884575 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:31:42.884546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:32:27.128784 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.128750    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5fvbm"]
Sep 16 18:32:27.131984 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.131966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5fvbm"
Sep 16 18:32:27.134678 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.134654    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:32:27.135560 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.135541    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:32:27.135560 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.135541    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xcgv4\""
Sep 16 18:32:27.142935 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.142916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5fvbm"]
Sep 16 18:32:27.279461 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.279417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z45pq\" (UniqueName: \"kubernetes.io/projected/afcb39a0-4ecf-4e97-8c3c-932ebf04bd64-kube-api-access-z45pq\") pod \"cert-manager-55b4d954d5-5fvbm\" (UID: \"afcb39a0-4ecf-4e97-8c3c-932ebf04bd64\") " pod="cert-manager/cert-manager-55b4d954d5-5fvbm"
Sep 16 18:32:27.279605 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.279490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afcb39a0-4ecf-4e97-8c3c-932ebf04bd64-bound-sa-token\") pod \"cert-manager-55b4d954d5-5fvbm\" (UID: \"afcb39a0-4ecf-4e97-8c3c-932ebf04bd64\") " pod="cert-manager/cert-manager-55b4d954d5-5fvbm"
Sep 16 18:32:27.379840 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.379781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afcb39a0-4ecf-4e97-8c3c-932ebf04bd64-bound-sa-token\") pod \"cert-manager-55b4d954d5-5fvbm\" (UID: \"afcb39a0-4ecf-4e97-8c3c-932ebf04bd64\") " pod="cert-manager/cert-manager-55b4d954d5-5fvbm"
Sep 16 18:32:27.379840 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.379831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z45pq\" (UniqueName: \"kubernetes.io/projected/afcb39a0-4ecf-4e97-8c3c-932ebf04bd64-kube-api-access-z45pq\") pod \"cert-manager-55b4d954d5-5fvbm\" (UID: \"afcb39a0-4ecf-4e97-8c3c-932ebf04bd64\") " pod="cert-manager/cert-manager-55b4d954d5-5fvbm"
Sep 16 18:32:27.389396 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.389379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z45pq\" (UniqueName: \"kubernetes.io/projected/afcb39a0-4ecf-4e97-8c3c-932ebf04bd64-kube-api-access-z45pq\") pod \"cert-manager-55b4d954d5-5fvbm\" (UID: \"afcb39a0-4ecf-4e97-8c3c-932ebf04bd64\") " pod="cert-manager/cert-manager-55b4d954d5-5fvbm"
Sep 16 18:32:27.389462 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.389394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afcb39a0-4ecf-4e97-8c3c-932ebf04bd64-bound-sa-token\") pod \"cert-manager-55b4d954d5-5fvbm\" (UID: \"afcb39a0-4ecf-4e97-8c3c-932ebf04bd64\") " pod="cert-manager/cert-manager-55b4d954d5-5fvbm"
Sep 16 18:32:27.440250 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.440225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5fvbm"
Sep 16 18:32:27.712801 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:27.712621    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5fvbm"]
Sep 16 18:32:27.715480 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:32:27.715452    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafcb39a0_4ecf_4e97_8c3c_932ebf04bd64.slice/crio-0ff1fd2f3c374c14c27374852ec9f796cc13d2d29c6fcdae47802ebbb83cad4f WatchSource:0}: Error finding container 0ff1fd2f3c374c14c27374852ec9f796cc13d2d29c6fcdae47802ebbb83cad4f: Status 404 returned error can't find the container with id 0ff1fd2f3c374c14c27374852ec9f796cc13d2d29c6fcdae47802ebbb83cad4f
Sep 16 18:32:28.027522 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:28.027487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5fvbm" event={"ID":"afcb39a0-4ecf-4e97-8c3c-932ebf04bd64","Type":"ContainerStarted","Data":"0ff1fd2f3c374c14c27374852ec9f796cc13d2d29c6fcdae47802ebbb83cad4f"}
Sep 16 18:32:31.041227 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:31.041192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5fvbm" event={"ID":"afcb39a0-4ecf-4e97-8c3c-932ebf04bd64","Type":"ContainerStarted","Data":"13c46ab59c8243d535480c4ffcea8a874fd8a741974cf2e6e72b63956f304d87"}
Sep 16 18:32:31.064037 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:31.063991    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-5fvbm" podStartSLOduration=1.079399097 podStartE2EDuration="4.063976996s" totalImagesPullingTime="2.984577899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:32:31.062807149 +0000 UTC m=+293.772046543" watchObservedRunningTime="2026-09-16 18:32:31.063976996 +0000 UTC m=+293.773216388"
Sep 16 18:32:37.689168 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:37.689138    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:32:37.692682 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:37.692655    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:32:37.700003 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:37.699980    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:32:49.451198 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.451164    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"]
Sep 16 18:32:49.454673 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.454622    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.460898 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.460877    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 18:32:49.461616 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.461591    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 18:32:49.462820 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.462803    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 18:32:49.465720 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.465700    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-srjkf\""
Sep 16 18:32:49.472786 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.472770    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 18:32:49.490974 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.490949    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"]
Sep 16 18:32:49.546275 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.546246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgvpr\" (UniqueName: \"kubernetes.io/projected/61e714c0-5317-47c8-ae86-623a63f55a5d-kube-api-access-jgvpr\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.546405 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.546278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61e714c0-5317-47c8-ae86-623a63f55a5d-webhook-cert\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.546405 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.546301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61e714c0-5317-47c8-ae86-623a63f55a5d-manifests\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.546405 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.546322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61e714c0-5317-47c8-ae86-623a63f55a5d-tmp\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.546544 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.546411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/61e714c0-5317-47c8-ae86-623a63f55a5d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.647717 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.647684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgvpr\" (UniqueName: \"kubernetes.io/projected/61e714c0-5317-47c8-ae86-623a63f55a5d-kube-api-access-jgvpr\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.647843 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.647718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61e714c0-5317-47c8-ae86-623a63f55a5d-webhook-cert\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.647843 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.647743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61e714c0-5317-47c8-ae86-623a63f55a5d-manifests\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.647843 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.647766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61e714c0-5317-47c8-ae86-623a63f55a5d-tmp\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.647843 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.647797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/61e714c0-5317-47c8-ae86-623a63f55a5d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.648103 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.648083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61e714c0-5317-47c8-ae86-623a63f55a5d-manifests\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.650192 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.650168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61e714c0-5317-47c8-ae86-623a63f55a5d-tmp\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.650288 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.650217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/61e714c0-5317-47c8-ae86-623a63f55a5d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.650368 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.650352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61e714c0-5317-47c8-ae86-623a63f55a5d-webhook-cert\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.659235 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.659216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgvpr\" (UniqueName: \"kubernetes.io/projected/61e714c0-5317-47c8-ae86-623a63f55a5d-kube-api-access-jgvpr\") pod \"opendatahub-operator-controller-manager-744584488c-2gtqt\" (UID: \"61e714c0-5317-47c8-ae86-623a63f55a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:49.765045 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:49.764988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:32:50.051097 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:50.051055    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"]
Sep 16 18:32:50.055302 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:32:50.055272    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e714c0_5317_47c8_ae86_623a63f55a5d.slice/crio-0d66afec60647c6df1f7c065cd8b9f14652c5ded2392026fc4dd3a0836047f5d WatchSource:0}: Error finding container 0d66afec60647c6df1f7c065cd8b9f14652c5ded2392026fc4dd3a0836047f5d: Status 404 returned error can't find the container with id 0d66afec60647c6df1f7c065cd8b9f14652c5ded2392026fc4dd3a0836047f5d
Sep 16 18:32:50.057157 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:50.057139    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:32:50.111867 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:50.111844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt" event={"ID":"61e714c0-5317-47c8-ae86-623a63f55a5d","Type":"ContainerStarted","Data":"0d66afec60647c6df1f7c065cd8b9f14652c5ded2392026fc4dd3a0836047f5d"}
Sep 16 18:32:53.125317 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:53.125284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt" event={"ID":"61e714c0-5317-47c8-ae86-623a63f55a5d","Type":"ContainerStarted","Data":"916c0291b1fba17e5d69bc0e8e76e38922de61b195917befdea708265ff92f25"}
Sep 16 18:32:54.131116 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:54.131081    2580 generic.go:356] "Generic (PLEG): container finished" podID="61e714c0-5317-47c8-ae86-623a63f55a5d" containerID="916c0291b1fba17e5d69bc0e8e76e38922de61b195917befdea708265ff92f25" exitCode=0
Sep 16 18:32:54.131456 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:54.131159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt" event={"ID":"61e714c0-5317-47c8-ae86-623a63f55a5d","Type":"ContainerDied","Data":"916c0291b1fba17e5d69bc0e8e76e38922de61b195917befdea708265ff92f25"}
Sep 16 18:32:55.136979 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:55.136941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt" event={"ID":"61e714c0-5317-47c8-ae86-623a63f55a5d","Type":"ContainerStarted","Data":"c649c4277c8c0cb75e48099fa53a90f1e8f2b46dcf1cb157947948a1675e16ff"}
Sep 16 18:32:55.137454 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:32:55.137074    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:33:06.142581 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:06.142544    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt"
Sep 16 18:33:06.171769 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:06.171717    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-744584488c-2gtqt" podStartSLOduration=12.796018042 podStartE2EDuration="17.171701011s" totalImagesPullingTime="3.316680455s" totalInitContainerRuntime="1.059002514s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:32:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:32:55.174563248 +0000 UTC m=+317.883802654" watchObservedRunningTime="2026-09-16 18:33:06.171701011 +0000 UTC m=+328.880940403"
Sep 16 18:33:09.114874 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.114838    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"]
Sep 16 18:33:09.116889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.116871    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"
Sep 16 18:33:09.124464 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.124431    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 18:33:09.124648 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.124624    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:33:09.124717 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.124624    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 18:33:09.124717 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.124664    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:33:09.154672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.154646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"]
Sep 16 18:33:09.212425 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.212396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcqh\" (UniqueName: \"kubernetes.io/projected/f923cd31-3318-4e76-8c1a-5c515f868aa6-kube-api-access-qfcqh\") pod \"kube-auth-proxy-84fbdbf5bc-mwsnr\" (UID: \"f923cd31-3318-4e76-8c1a-5c515f868aa6\") " pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"
Sep 16 18:33:09.212425 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.212427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f923cd31-3318-4e76-8c1a-5c515f868aa6-tmp\") pod \"kube-auth-proxy-84fbdbf5bc-mwsnr\" (UID: \"f923cd31-3318-4e76-8c1a-5c515f868aa6\") " pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"
Sep 16 18:33:09.212588 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.212454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f923cd31-3318-4e76-8c1a-5c515f868aa6-tls-certs\") pod \"kube-auth-proxy-84fbdbf5bc-mwsnr\" (UID: \"f923cd31-3318-4e76-8c1a-5c515f868aa6\") " pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"
Sep 16 18:33:09.313668 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.313636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcqh\" (UniqueName: \"kubernetes.io/projected/f923cd31-3318-4e76-8c1a-5c515f868aa6-kube-api-access-qfcqh\") pod \"kube-auth-proxy-84fbdbf5bc-mwsnr\" (UID: \"f923cd31-3318-4e76-8c1a-5c515f868aa6\") " pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"
Sep 16 18:33:09.313668 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.313670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f923cd31-3318-4e76-8c1a-5c515f868aa6-tmp\") pod \"kube-auth-proxy-84fbdbf5bc-mwsnr\" (UID: \"f923cd31-3318-4e76-8c1a-5c515f868aa6\") " pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"
Sep 16 18:33:09.313834 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.313704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f923cd31-3318-4e76-8c1a-5c515f868aa6-tls-certs\") pod \"kube-auth-proxy-84fbdbf5bc-mwsnr\" (UID: \"f923cd31-3318-4e76-8c1a-5c515f868aa6\") " pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"
Sep 16 18:33:09.315932 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.315907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f923cd31-3318-4e76-8c1a-5c515f868aa6-tmp\") pod \"kube-auth-proxy-84fbdbf5bc-mwsnr\" (UID: \"f923cd31-3318-4e76-8c1a-5c515f868aa6\") " pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"
Sep 16 18:33:09.316063 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.316051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f923cd31-3318-4e76-8c1a-5c515f868aa6-tls-certs\") pod \"kube-auth-proxy-84fbdbf5bc-mwsnr\" (UID: \"f923cd31-3318-4e76-8c1a-5c515f868aa6\") " pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"
Sep 16 18:33:09.331832 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.331802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcqh\" (UniqueName: \"kubernetes.io/projected/f923cd31-3318-4e76-8c1a-5c515f868aa6-kube-api-access-qfcqh\") pod \"kube-auth-proxy-84fbdbf5bc-mwsnr\" (UID: \"f923cd31-3318-4e76-8c1a-5c515f868aa6\") " pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"
Sep 16 18:33:09.428736 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.428652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"
Sep 16 18:33:09.535144 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.535108    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"]
Sep 16 18:33:09.537251 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.537236    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.542138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.542115    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:33:09.542253 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.542236    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mm8sp\""
Sep 16 18:33:09.542493 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.542480    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:33:09.543208 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.543188    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:33:09.543286 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.543245    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:33:09.543356 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.543286    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:33:09.565353 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.565308    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"]
Sep 16 18:33:09.616643 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.616611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fd308eb7-6a7f-4b70-8dae-4860a238b91d-manager-config\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.616643 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.616640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd308eb7-6a7f-4b70-8dae-4860a238b91d-cert\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.616818 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.616662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nqmq\" (UniqueName: \"kubernetes.io/projected/fd308eb7-6a7f-4b70-8dae-4860a238b91d-kube-api-access-7nqmq\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.616818 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.616763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd308eb7-6a7f-4b70-8dae-4860a238b91d-metrics-cert\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.714072 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.713987    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr"]
Sep 16 18:33:09.716738 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:33:09.716712    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf923cd31_3318_4e76_8c1a_5c515f868aa6.slice/crio-4678e8df0578e28bc9a0fc3409fbcb02c49c1bab9ba073d472403530e2c55240 WatchSource:0}: Error finding container 4678e8df0578e28bc9a0fc3409fbcb02c49c1bab9ba073d472403530e2c55240: Status 404 returned error can't find the container with id 4678e8df0578e28bc9a0fc3409fbcb02c49c1bab9ba073d472403530e2c55240
Sep 16 18:33:09.717641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.717616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd308eb7-6a7f-4b70-8dae-4860a238b91d-metrics-cert\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.717752 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.717665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fd308eb7-6a7f-4b70-8dae-4860a238b91d-manager-config\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.717752 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.717682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd308eb7-6a7f-4b70-8dae-4860a238b91d-cert\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.717752 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.717704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nqmq\" (UniqueName: \"kubernetes.io/projected/fd308eb7-6a7f-4b70-8dae-4860a238b91d-kube-api-access-7nqmq\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.718525 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.718402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fd308eb7-6a7f-4b70-8dae-4860a238b91d-manager-config\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.720385 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.720351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd308eb7-6a7f-4b70-8dae-4860a238b91d-cert\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.720477 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.720393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd308eb7-6a7f-4b70-8dae-4860a238b91d-metrics-cert\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.729140 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.729116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nqmq\" (UniqueName: \"kubernetes.io/projected/fd308eb7-6a7f-4b70-8dae-4860a238b91d-kube-api-access-7nqmq\") pod \"lws-controller-manager-75b97559b-w4plt\" (UID: \"fd308eb7-6a7f-4b70-8dae-4860a238b91d\") " pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:09.846617 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:09.846590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:10.138282 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:10.138243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"]
Sep 16 18:33:10.141710 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:33:10.141684    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd308eb7_6a7f_4b70_8dae_4860a238b91d.slice/crio-5c44320a8d54b372543a7acd491f57a5bbcc868af0d7bf8ac7c6d4dbb6465442 WatchSource:0}: Error finding container 5c44320a8d54b372543a7acd491f57a5bbcc868af0d7bf8ac7c6d4dbb6465442: Status 404 returned error can't find the container with id 5c44320a8d54b372543a7acd491f57a5bbcc868af0d7bf8ac7c6d4dbb6465442
Sep 16 18:33:10.194656 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:10.194629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt" event={"ID":"fd308eb7-6a7f-4b70-8dae-4860a238b91d","Type":"ContainerStarted","Data":"5c44320a8d54b372543a7acd491f57a5bbcc868af0d7bf8ac7c6d4dbb6465442"}
Sep 16 18:33:10.195808 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:10.195787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr" event={"ID":"f923cd31-3318-4e76-8c1a-5c515f868aa6","Type":"ContainerStarted","Data":"4678e8df0578e28bc9a0fc3409fbcb02c49c1bab9ba073d472403530e2c55240"}
Sep 16 18:33:14.219527 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:14.219482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr" event={"ID":"f923cd31-3318-4e76-8c1a-5c515f868aa6","Type":"ContainerStarted","Data":"d273b030e543dbeeea97f387fd81c4536586c23ff583277b516654cd962b4549"}
Sep 16 18:33:14.220997 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:14.220967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt" event={"ID":"fd308eb7-6a7f-4b70-8dae-4860a238b91d","Type":"ContainerStarted","Data":"13cb5a4c604ff578c7cb070dd1fb6ffa9bbbaf98aa24424c11087167b34c5d47"}
Sep 16 18:33:14.221131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:14.221029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:33:14.251394 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:14.251325    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84fbdbf5bc-mwsnr" podStartSLOduration=1.258174873 podStartE2EDuration="5.251307827s" totalImagesPullingTime="3.993132954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:33:14.249022035 +0000 UTC m=+336.958261426" watchObservedRunningTime="2026-09-16 18:33:14.251307827 +0000 UTC m=+336.960547218"
Sep 16 18:33:14.288067 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:14.288012    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt" podStartSLOduration=1.672721447 podStartE2EDuration="5.287996475s" totalImagesPullingTime="3.615275028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:33:14.286495516 +0000 UTC m=+336.995734910" watchObservedRunningTime="2026-09-16 18:33:14.287996475 +0000 UTC m=+336.997235867"
Sep 16 18:33:25.228116 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:33:25.228082    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-75b97559b-w4plt"
Sep 16 18:35:13.165717 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.165639    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"]
Sep 16 18:35:13.168831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.168813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:35:13.172536 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.172513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-s74nn\""
Sep 16 18:35:13.173414 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.173392    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:35:13.173513 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.173425    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:35:13.235791 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.235757    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"]
Sep 16 18:35:13.237518 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.237496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92bc46d0-34a8-44aa-97d8-dd13c100d25d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-997bc\" (UID: \"92bc46d0-34a8-44aa-97d8-dd13c100d25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:35:13.237623 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.237527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9s7\" (UniqueName: \"kubernetes.io/projected/92bc46d0-34a8-44aa-97d8-dd13c100d25d-kube-api-access-lc9s7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-997bc\" (UID: \"92bc46d0-34a8-44aa-97d8-dd13c100d25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:35:13.339007 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.338975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92bc46d0-34a8-44aa-97d8-dd13c100d25d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-997bc\" (UID: \"92bc46d0-34a8-44aa-97d8-dd13c100d25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:35:13.339159 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.339014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9s7\" (UniqueName: \"kubernetes.io/projected/92bc46d0-34a8-44aa-97d8-dd13c100d25d-kube-api-access-lc9s7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-997bc\" (UID: \"92bc46d0-34a8-44aa-97d8-dd13c100d25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:35:13.339307 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.339284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92bc46d0-34a8-44aa-97d8-dd13c100d25d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-997bc\" (UID: \"92bc46d0-34a8-44aa-97d8-dd13c100d25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:35:13.366448 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.366423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9s7\" (UniqueName: \"kubernetes.io/projected/92bc46d0-34a8-44aa-97d8-dd13c100d25d-kube-api-access-lc9s7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-997bc\" (UID: \"92bc46d0-34a8-44aa-97d8-dd13c100d25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:35:13.479447 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.479378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:35:13.764174 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:13.764148    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"]
Sep 16 18:35:13.766433 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:35:13.766403    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92bc46d0_34a8_44aa_97d8_dd13c100d25d.slice/crio-cbd952005638e36208f0ab0caeec6e438255624667f77225ab253c9e8cf40b60 WatchSource:0}: Error finding container cbd952005638e36208f0ab0caeec6e438255624667f77225ab253c9e8cf40b60: Status 404 returned error can't find the container with id cbd952005638e36208f0ab0caeec6e438255624667f77225ab253c9e8cf40b60
Sep 16 18:35:14.710539 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:14.710497    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc" event={"ID":"92bc46d0-34a8-44aa-97d8-dd13c100d25d","Type":"ContainerStarted","Data":"cbd952005638e36208f0ab0caeec6e438255624667f77225ab253c9e8cf40b60"}
Sep 16 18:35:17.728636 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:17.728551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc" event={"ID":"92bc46d0-34a8-44aa-97d8-dd13c100d25d","Type":"ContainerStarted","Data":"6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c"}
Sep 16 18:35:17.728966 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:17.728669    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:35:17.797033 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:17.796975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc" podStartSLOduration=2.214045078 podStartE2EDuration="5.796956114s" totalImagesPullingTime="3.582911036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:17.796060857 +0000 UTC m=+460.505300249" watchObservedRunningTime="2026-09-16 18:35:17.796956114 +0000 UTC m=+460.506195508"
Sep 16 18:35:28.735479 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:28.735450    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:35:49.022515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.022480    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:35:49.026774 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.026755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"
Sep 16 18:35:49.029750 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.029727    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:35:49.029870 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.029837    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wcm6d\""
Sep 16 18:35:49.047248 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.047224    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:35:49.132817 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.132784    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5fffb0aa-a7bd-4c8c-8660-bd3079b73181-config-file\") pod \"limitador-limitador-84994f7ffc-v9l7j\" (UID: \"5fffb0aa-a7bd-4c8c-8660-bd3079b73181\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"
Sep 16 18:35:49.132973 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.132851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqvn2\" (UniqueName: \"kubernetes.io/projected/5fffb0aa-a7bd-4c8c-8660-bd3079b73181-kube-api-access-lqvn2\") pod \"limitador-limitador-84994f7ffc-v9l7j\" (UID: \"5fffb0aa-a7bd-4c8c-8660-bd3079b73181\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"
Sep 16 18:35:49.153306 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.153280    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:35:49.233633 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.233599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqvn2\" (UniqueName: \"kubernetes.io/projected/5fffb0aa-a7bd-4c8c-8660-bd3079b73181-kube-api-access-lqvn2\") pod \"limitador-limitador-84994f7ffc-v9l7j\" (UID: \"5fffb0aa-a7bd-4c8c-8660-bd3079b73181\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"
Sep 16 18:35:49.233777 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.233693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5fffb0aa-a7bd-4c8c-8660-bd3079b73181-config-file\") pod \"limitador-limitador-84994f7ffc-v9l7j\" (UID: \"5fffb0aa-a7bd-4c8c-8660-bd3079b73181\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"
Sep 16 18:35:49.234314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.234290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5fffb0aa-a7bd-4c8c-8660-bd3079b73181-config-file\") pod \"limitador-limitador-84994f7ffc-v9l7j\" (UID: \"5fffb0aa-a7bd-4c8c-8660-bd3079b73181\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"
Sep 16 18:35:49.253213 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.253190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqvn2\" (UniqueName: \"kubernetes.io/projected/5fffb0aa-a7bd-4c8c-8660-bd3079b73181-kube-api-access-lqvn2\") pod \"limitador-limitador-84994f7ffc-v9l7j\" (UID: \"5fffb0aa-a7bd-4c8c-8660-bd3079b73181\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"
Sep 16 18:35:49.336053 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.335988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"
Sep 16 18:35:49.612737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.612710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:35:49.614746 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:35:49.614719    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fffb0aa_a7bd_4c8c_8660_bd3079b73181.slice/crio-cf6b1288ecf894c13e82fb60a38eae5415539ab7457c9018a8bea0ab8e0cd50a WatchSource:0}: Error finding container cf6b1288ecf894c13e82fb60a38eae5415539ab7457c9018a8bea0ab8e0cd50a: Status 404 returned error can't find the container with id cf6b1288ecf894c13e82fb60a38eae5415539ab7457c9018a8bea0ab8e0cd50a
Sep 16 18:35:49.858239 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.858202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j" event={"ID":"5fffb0aa-a7bd-4c8c-8660-bd3079b73181","Type":"ContainerStarted","Data":"cf6b1288ecf894c13e82fb60a38eae5415539ab7457c9018a8bea0ab8e0cd50a"}
Sep 16 18:35:49.937277 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.937200    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-dbkgw"]
Sep 16 18:35:49.939975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.939954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dbkgw"
Sep 16 18:35:49.942807 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.942785    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gchll\""
Sep 16 18:35:49.949772 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:49.949750    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dbkgw"]
Sep 16 18:35:50.040161 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:50.040132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkzgz\" (UniqueName: \"kubernetes.io/projected/d3ca5989-6416-446b-a5bd-39470b5a5445-kube-api-access-jkzgz\") pod \"authorino-59c974bd48-dbkgw\" (UID: \"d3ca5989-6416-446b-a5bd-39470b5a5445\") " pod="kuadrant-system/authorino-59c974bd48-dbkgw"
Sep 16 18:35:50.140778 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:50.140748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkzgz\" (UniqueName: \"kubernetes.io/projected/d3ca5989-6416-446b-a5bd-39470b5a5445-kube-api-access-jkzgz\") pod \"authorino-59c974bd48-dbkgw\" (UID: \"d3ca5989-6416-446b-a5bd-39470b5a5445\") " pod="kuadrant-system/authorino-59c974bd48-dbkgw"
Sep 16 18:35:50.153148 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:50.153120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkzgz\" (UniqueName: \"kubernetes.io/projected/d3ca5989-6416-446b-a5bd-39470b5a5445-kube-api-access-jkzgz\") pod \"authorino-59c974bd48-dbkgw\" (UID: \"d3ca5989-6416-446b-a5bd-39470b5a5445\") " pod="kuadrant-system/authorino-59c974bd48-dbkgw"
Sep 16 18:35:50.249308 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:50.249225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dbkgw"
Sep 16 18:35:50.599301 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:50.599269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dbkgw"]
Sep 16 18:35:50.601669 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:35:50.601637    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3ca5989_6416_446b_a5bd_39470b5a5445.slice/crio-667b4200fdc61e7aa709d6f67a65a3e0b914fa61cf4022cbbbd57b80ebf5fe8a WatchSource:0}: Error finding container 667b4200fdc61e7aa709d6f67a65a3e0b914fa61cf4022cbbbd57b80ebf5fe8a: Status 404 returned error can't find the container with id 667b4200fdc61e7aa709d6f67a65a3e0b914fa61cf4022cbbbd57b80ebf5fe8a
Sep 16 18:35:50.863606 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:50.863514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dbkgw" event={"ID":"d3ca5989-6416-446b-a5bd-39470b5a5445","Type":"ContainerStarted","Data":"667b4200fdc61e7aa709d6f67a65a3e0b914fa61cf4022cbbbd57b80ebf5fe8a"}
Sep 16 18:35:50.865020 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:50.864991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j" event={"ID":"5fffb0aa-a7bd-4c8c-8660-bd3079b73181","Type":"ContainerStarted","Data":"e2de0d2d113b559101b3426b1b2f7ed2896feba488851c73723f708162913954"}
Sep 16 18:35:50.865155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:50.865120    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"
Sep 16 18:35:50.898443 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:50.898400    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j" podStartSLOduration=2.151163244 podStartE2EDuration="2.898385039s" totalImagesPullingTime="747.221795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:50.89721793 +0000 UTC m=+493.606457323" watchObservedRunningTime="2026-09-16 18:35:50.898385039 +0000 UTC m=+493.607624430"
Sep 16 18:35:52.874965 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:52.874928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dbkgw" event={"ID":"d3ca5989-6416-446b-a5bd-39470b5a5445","Type":"ContainerStarted","Data":"c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75"}
Sep 16 18:35:52.900076 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:35:52.900025    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-dbkgw" podStartSLOduration=2.238266304 podStartE2EDuration="3.899988205s" totalImagesPullingTime="1.661721901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:52.898465045 +0000 UTC m=+495.607704436" watchObservedRunningTime="2026-09-16 18:35:52.899988205 +0000 UTC m=+495.609227596"
Sep 16 18:36:01.870960 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:01.870924    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"
Sep 16 18:36:20.161381 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:20.161351    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dbkgw"]
Sep 16 18:36:20.161889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:20.161531    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-dbkgw" podUID="d3ca5989-6416-446b-a5bd-39470b5a5445" containerName="authorino" containerID="cri-o://c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75" gracePeriod=30
Sep 16 18:36:20.397610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:20.397587    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dbkgw"
Sep 16 18:36:20.504494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:20.504400    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkzgz\" (UniqueName: \"kubernetes.io/projected/d3ca5989-6416-446b-a5bd-39470b5a5445-kube-api-access-jkzgz\") pod \"d3ca5989-6416-446b-a5bd-39470b5a5445\" (UID: \"d3ca5989-6416-446b-a5bd-39470b5a5445\") "
Sep 16 18:36:20.506553 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:20.506523    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ca5989-6416-446b-a5bd-39470b5a5445-kube-api-access-jkzgz" (OuterVolumeSpecName: "kube-api-access-jkzgz") pod "d3ca5989-6416-446b-a5bd-39470b5a5445" (UID: "d3ca5989-6416-446b-a5bd-39470b5a5445"). InnerVolumeSpecName "kube-api-access-jkzgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:36:20.605319 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:20.605289    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkzgz\" (UniqueName: \"kubernetes.io/projected/d3ca5989-6416-446b-a5bd-39470b5a5445-kube-api-access-jkzgz\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:36:21.007180 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:21.007142    2580 generic.go:356] "Generic (PLEG): container finished" podID="d3ca5989-6416-446b-a5bd-39470b5a5445" containerID="c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75" exitCode=0
Sep 16 18:36:21.007398 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:21.007186    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dbkgw"
Sep 16 18:36:21.007398 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:21.007229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dbkgw" event={"ID":"d3ca5989-6416-446b-a5bd-39470b5a5445","Type":"ContainerDied","Data":"c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75"}
Sep 16 18:36:21.007398 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:21.007278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dbkgw" event={"ID":"d3ca5989-6416-446b-a5bd-39470b5a5445","Type":"ContainerDied","Data":"667b4200fdc61e7aa709d6f67a65a3e0b914fa61cf4022cbbbd57b80ebf5fe8a"}
Sep 16 18:36:21.007398 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:21.007301    2580 scope.go:117] "RemoveContainer" containerID="c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75"
Sep 16 18:36:21.015496 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:21.015476    2580 scope.go:117] "RemoveContainer" containerID="c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75"
Sep 16 18:36:21.015811 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:36:21.015789    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75\": container with ID starting with c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75 not found: ID does not exist" containerID="c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75"
Sep 16 18:36:21.015878 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:21.015819    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75"} err="failed to get container status \"c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75\": rpc error: code = NotFound desc = could not find container \"c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75\": container with ID starting with c384b413a03fb7d292a9153f88d673bf26b3088e0f6f1b27c4a740596dd6ad75 not found: ID does not exist"
Sep 16 18:36:21.050977 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:21.050947    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dbkgw"]
Sep 16 18:36:21.053041 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:21.053018    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dbkgw"]
Sep 16 18:36:21.782699 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:36:21.782667    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3ca5989-6416-446b-a5bd-39470b5a5445" path="/var/lib/kubelet/pods/d3ca5989-6416-446b-a5bd-39470b5a5445/volumes"
Sep 16 18:37:03.855317 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:03.855280    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:37:25.700193 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:25.700162    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:37:34.045699 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:34.045662    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:37:37.719003 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:37.718973    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:37:37.721521 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:37.721499    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:37:39.626492 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.626458    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"]
Sep 16 18:37:39.626953 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.626790    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ca5989-6416-446b-a5bd-39470b5a5445" containerName="authorino"
Sep 16 18:37:39.626953 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.626802    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ca5989-6416-446b-a5bd-39470b5a5445" containerName="authorino"
Sep 16 18:37:39.626953 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.626871    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ca5989-6416-446b-a5bd-39470b5a5445" containerName="authorino"
Sep 16 18:37:39.628706 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.628687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.632910 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.632883    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 18:37:39.633089 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.633067    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 18:37:39.633148 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.633067    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4pg6k\""
Sep 16 18:37:39.633190 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.633156    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 18:37:39.646954 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.646929    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"]
Sep 16 18:37:39.749127 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.749089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.749286 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.749137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.749286 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.749228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.749286 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.749260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-467nx\" (UniqueName: \"kubernetes.io/projected/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-kube-api-access-467nx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.749455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.749294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.749455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.749356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.850371 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.850344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.850540 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.850384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-467nx\" (UniqueName: \"kubernetes.io/projected/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-kube-api-access-467nx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.850540 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.850414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.850540 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.850430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.850540 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.850458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.850540 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.850483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.850802 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.850711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.850802 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.850763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.850894 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.850826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.852775 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.852749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.852951 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.852932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.860457 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.860437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-467nx\" (UniqueName: \"kubernetes.io/projected/f2a7cb9f-9714-4467-bd76-9ce0b9b23733-kube-api-access-467nx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w\" (UID: \"f2a7cb9f-9714-4467-bd76-9ce0b9b23733\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:39.939453 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:39.939385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:40.226096 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:40.226067    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"]
Sep 16 18:37:40.228252 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:37:40.228223    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a7cb9f_9714_4467_bd76_9ce0b9b23733.slice/crio-6b5870391d82721d5db85927e289a38a2f61b8dea2a4ca389405abe6429b6006 WatchSource:0}: Error finding container 6b5870391d82721d5db85927e289a38a2f61b8dea2a4ca389405abe6429b6006: Status 404 returned error can't find the container with id 6b5870391d82721d5db85927e289a38a2f61b8dea2a4ca389405abe6429b6006
Sep 16 18:37:40.342032 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:40.341997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w" event={"ID":"f2a7cb9f-9714-4467-bd76-9ce0b9b23733","Type":"ContainerStarted","Data":"6b5870391d82721d5db85927e289a38a2f61b8dea2a4ca389405abe6429b6006"}
Sep 16 18:37:40.925986 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:40.925949    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:37:41.348295 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:41.348251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w" event={"ID":"f2a7cb9f-9714-4467-bd76-9ce0b9b23733","Type":"ContainerStarted","Data":"5e50bedf15c3feecaee3bf151bd975c0c6faafef382e2631440f8374f5dfdfcc"}
Sep 16 18:37:41.371061 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:41.371000    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w" podStartSLOduration=1.386546732 podStartE2EDuration="2.370983299s" totalImagesPullingTime="984.436567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:37:41.369182555 +0000 UTC m=+604.078421949" watchObservedRunningTime="2026-09-16 18:37:41.370983299 +0000 UTC m=+604.080222723"
Sep 16 18:37:49.939788 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:49.939755    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:49.941164 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:49.939819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:49.951929 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:49.951899    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:37:50.396832 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:37:50.396794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w"
Sep 16 18:38:06.337904 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:06.337826    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:38:15.018106 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.018066    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"]
Sep 16 18:38:15.024434 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.024413    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.026832 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.026814    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 18:38:15.034775 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.034752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"]
Sep 16 18:38:15.148831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.148796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjjmt\" (UniqueName: \"kubernetes.io/projected/45e0ca04-996d-4c0f-a928-312369224450-kube-api-access-fjjmt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.148976 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.148844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.148976 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.148893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.148976 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.148924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.149115 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.148990    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.149115 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.149012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45e0ca04-996d-4c0f-a928-312369224450-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.250092 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.250036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.250092 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.250096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.250384 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.250123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.250384 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.250157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.250384 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.250187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45e0ca04-996d-4c0f-a928-312369224450-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.250384 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.250283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjjmt\" (UniqueName: \"kubernetes.io/projected/45e0ca04-996d-4c0f-a928-312369224450-kube-api-access-fjjmt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.250600 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.250529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.250600 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.250534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.250697 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.250599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.252696 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.252674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45e0ca04-996d-4c0f-a928-312369224450-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.252696 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.252688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45e0ca04-996d-4c0f-a928-312369224450-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.260889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.260868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjjmt\" (UniqueName: \"kubernetes.io/projected/45e0ca04-996d-4c0f-a928-312369224450-kube-api-access-fjjmt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn\" (UID: \"45e0ca04-996d-4c0f-a928-312369224450\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.333960 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.333911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:15.611252 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.611223    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"]
Sep 16 18:38:15.613463 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:38:15.613401    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e0ca04_996d_4c0f_a928_312369224450.slice/crio-ad6a49889b58abd20b41aa6af1e62cce5dab5a1c251ad5f47631899cea0fe3b8 WatchSource:0}: Error finding container ad6a49889b58abd20b41aa6af1e62cce5dab5a1c251ad5f47631899cea0fe3b8: Status 404 returned error can't find the container with id ad6a49889b58abd20b41aa6af1e62cce5dab5a1c251ad5f47631899cea0fe3b8
Sep 16 18:38:15.614771 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:15.614751    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:38:16.500815 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:16.500775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn" event={"ID":"45e0ca04-996d-4c0f-a928-312369224450","Type":"ContainerStarted","Data":"66d4f87913abc94b368eb854035e3c385f404ac6ac63c62f08dbe06ea9d1b323"}
Sep 16 18:38:16.501184 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:16.500852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn" event={"ID":"45e0ca04-996d-4c0f-a928-312369224450","Type":"ContainerStarted","Data":"ad6a49889b58abd20b41aa6af1e62cce5dab5a1c251ad5f47631899cea0fe3b8"}
Sep 16 18:38:16.522255 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:16.522204    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn" podStartSLOduration=2.34365327 podStartE2EDuration="2.522189573s" totalImagesPullingTime="178.536303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:38:16.519719255 +0000 UTC m=+639.228958646" watchObservedRunningTime="2026-09-16 18:38:16.522189573 +0000 UTC m=+639.231428965"
Sep 16 18:38:16.525903 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:16.525879    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:38:25.334938 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:25.334890    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:25.335317 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:25.334966    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:25.347603 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:25.347580    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:25.555534 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:25.555498    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn"
Sep 16 18:38:28.141917 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:28.141875    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:38:45.352792 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.352762    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"]
Sep 16 18:38:45.355759 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.355741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.361753 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.361656    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 18:38:45.376162 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.376135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.376306 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.376186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.376306 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.376237    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"]
Sep 16 18:38:45.376306 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.376272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.376494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.376310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.376494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.376356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.376494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.376402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv4pp\" (UniqueName: \"kubernetes.io/projected/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-kube-api-access-tv4pp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.476783 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.476746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.476961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.476807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.476961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.476836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.476961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.476868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.476961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.476917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4pp\" (UniqueName: \"kubernetes.io/projected/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-kube-api-access-tv4pp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.476961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.476947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.477225 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.477211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.477286 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.477253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.477370 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.477302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.479224 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.479202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.479427 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.479403    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.490493 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.490469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv4pp\" (UniqueName: \"kubernetes.io/projected/7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0-kube-api-access-tv4pp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw\" (UID: \"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.664543 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.664466    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:45.945631 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:45.945600    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"]
Sep 16 18:38:45.948268 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:38:45.948236    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b874b7a_cfb2_4d8c_8ece_a804cd4be3c0.slice/crio-13426d541665834937a0c84d88c5785fc38bcc457b8c4d5344ef522b7956030d WatchSource:0}: Error finding container 13426d541665834937a0c84d88c5785fc38bcc457b8c4d5344ef522b7956030d: Status 404 returned error can't find the container with id 13426d541665834937a0c84d88c5785fc38bcc457b8c4d5344ef522b7956030d
Sep 16 18:38:46.642743 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:46.642709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw" event={"ID":"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0","Type":"ContainerStarted","Data":"92cad1b02de8c308bf4d8557076ee6be2d5299732666ed6aec06bbbd274adcf2"}
Sep 16 18:38:46.643165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:46.642752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw" event={"ID":"7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0","Type":"ContainerStarted","Data":"13426d541665834937a0c84d88c5785fc38bcc457b8c4d5344ef522b7956030d"}
Sep 16 18:38:46.664615 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:46.664563    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw" podStartSLOduration=1.492843242 podStartE2EDuration="1.664542716s" totalImagesPullingTime="171.699474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:38:46.664061272 +0000 UTC m=+669.373300676" watchObservedRunningTime="2026-09-16 18:38:46.664542716 +0000 UTC m=+669.373782109"
Sep 16 18:38:47.636735 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:47.636702    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:38:55.664683 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:55.664652    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:55.664683 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:55.664694    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:55.677528 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:55.677501    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:38:55.693012 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:38:55.692988    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw"
Sep 16 18:40:15.063783 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.063747    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b47bf9dd7-dsq4b"]
Sep 16 18:40:15.066928 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.066908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b47bf9dd7-dsq4b"
Sep 16 18:40:15.070264 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.070242    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gchll\""
Sep 16 18:40:15.070395 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.070244    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 18:40:15.079768 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.079749    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b47bf9dd7-dsq4b"]
Sep 16 18:40:15.220896 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.220859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05469e3c-de98-43ac-a7a4-d8eac0bd013b-tls-cert\") pod \"authorino-5b47bf9dd7-dsq4b\" (UID: \"05469e3c-de98-43ac-a7a4-d8eac0bd013b\") " pod="kuadrant-system/authorino-5b47bf9dd7-dsq4b"
Sep 16 18:40:15.221074 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.220905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxt6\" (UniqueName: \"kubernetes.io/projected/05469e3c-de98-43ac-a7a4-d8eac0bd013b-kube-api-access-kdxt6\") pod \"authorino-5b47bf9dd7-dsq4b\" (UID: \"05469e3c-de98-43ac-a7a4-d8eac0bd013b\") " pod="kuadrant-system/authorino-5b47bf9dd7-dsq4b"
Sep 16 18:40:15.322135 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.322046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05469e3c-de98-43ac-a7a4-d8eac0bd013b-tls-cert\") pod \"authorino-5b47bf9dd7-dsq4b\" (UID: \"05469e3c-de98-43ac-a7a4-d8eac0bd013b\") " pod="kuadrant-system/authorino-5b47bf9dd7-dsq4b"
Sep 16 18:40:15.322135 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.322094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxt6\" (UniqueName: \"kubernetes.io/projected/05469e3c-de98-43ac-a7a4-d8eac0bd013b-kube-api-access-kdxt6\") pod \"authorino-5b47bf9dd7-dsq4b\" (UID: \"05469e3c-de98-43ac-a7a4-d8eac0bd013b\") " pod="kuadrant-system/authorino-5b47bf9dd7-dsq4b"
Sep 16 18:40:15.324564 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.324536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05469e3c-de98-43ac-a7a4-d8eac0bd013b-tls-cert\") pod \"authorino-5b47bf9dd7-dsq4b\" (UID: \"05469e3c-de98-43ac-a7a4-d8eac0bd013b\") " pod="kuadrant-system/authorino-5b47bf9dd7-dsq4b"
Sep 16 18:40:15.332088 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.332060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxt6\" (UniqueName: \"kubernetes.io/projected/05469e3c-de98-43ac-a7a4-d8eac0bd013b-kube-api-access-kdxt6\") pod \"authorino-5b47bf9dd7-dsq4b\" (UID: \"05469e3c-de98-43ac-a7a4-d8eac0bd013b\") " pod="kuadrant-system/authorino-5b47bf9dd7-dsq4b"
Sep 16 18:40:15.375614 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.375587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b47bf9dd7-dsq4b"
Sep 16 18:40:15.663131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:15.663059    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b47bf9dd7-dsq4b"]
Sep 16 18:40:15.666427 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:40:15.666394    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05469e3c_de98_43ac_a7a4_d8eac0bd013b.slice/crio-cc68d9878f1c2c176556ad7b7bcd98d14dc482faa34507c2a8d20e1acec2a44a WatchSource:0}: Error finding container cc68d9878f1c2c176556ad7b7bcd98d14dc482faa34507c2a8d20e1acec2a44a: Status 404 returned error can't find the container with id cc68d9878f1c2c176556ad7b7bcd98d14dc482faa34507c2a8d20e1acec2a44a
Sep 16 18:40:16.027463 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:16.027428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b47bf9dd7-dsq4b" event={"ID":"05469e3c-de98-43ac-a7a4-d8eac0bd013b","Type":"ContainerStarted","Data":"cc68d9878f1c2c176556ad7b7bcd98d14dc482faa34507c2a8d20e1acec2a44a"}
Sep 16 18:40:17.033275 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:17.033232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b47bf9dd7-dsq4b" event={"ID":"05469e3c-de98-43ac-a7a4-d8eac0bd013b","Type":"ContainerStarted","Data":"a809d15b86736ec24d1130f544ed0e28974ad2e8c3997b43ab807b79e65b8349"}
Sep 16 18:40:17.069837 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:17.069778    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b47bf9dd7-dsq4b" podStartSLOduration=0.895504651 podStartE2EDuration="2.069759337s" totalImagesPullingTime="1.174254686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:17.067975182 +0000 UTC m=+759.777214574" watchObservedRunningTime="2026-09-16 18:40:17.069759337 +0000 UTC m=+759.778998730"
Sep 16 18:40:41.022659 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:41.022622    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:40:43.183479 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:43.183448    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7648-49ccb2"]
Sep 16 18:40:43.186813 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:43.186793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-49ccb2"
Sep 16 18:40:43.189429 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:43.189410    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4g8kx\""
Sep 16 18:40:43.216137 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:43.216116    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-49ccb2"]
Sep 16 18:40:43.259248 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:43.259217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5tk\" (UniqueName: \"kubernetes.io/projected/bff71746-5b69-4358-be30-72ec1d839204-kube-api-access-6f5tk\") pod \"test-curl-7648-49ccb2\" (UID: \"bff71746-5b69-4358-be30-72ec1d839204\") " pod="opendatahub/test-curl-7648-49ccb2"
Sep 16 18:40:43.360252 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:43.360215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5tk\" (UniqueName: \"kubernetes.io/projected/bff71746-5b69-4358-be30-72ec1d839204-kube-api-access-6f5tk\") pod \"test-curl-7648-49ccb2\" (UID: \"bff71746-5b69-4358-be30-72ec1d839204\") " pod="opendatahub/test-curl-7648-49ccb2"
Sep 16 18:40:43.374636 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:43.374600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5tk\" (UniqueName: \"kubernetes.io/projected/bff71746-5b69-4358-be30-72ec1d839204-kube-api-access-6f5tk\") pod \"test-curl-7648-49ccb2\" (UID: \"bff71746-5b69-4358-be30-72ec1d839204\") " pod="opendatahub/test-curl-7648-49ccb2"
Sep 16 18:40:43.496573 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:43.496483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-49ccb2"
Sep 16 18:40:43.785509 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:43.785478    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-49ccb2"]
Sep 16 18:40:43.788750 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:40:43.788717    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbff71746_5b69_4358_be30_72ec1d839204.slice/crio-0c49780be854334d79c11f70ed9959f6612304fc424da5254670231c56b38b3d WatchSource:0}: Error finding container 0c49780be854334d79c11f70ed9959f6612304fc424da5254670231c56b38b3d: Status 404 returned error can't find the container with id 0c49780be854334d79c11f70ed9959f6612304fc424da5254670231c56b38b3d
Sep 16 18:40:44.152464 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:44.152380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-49ccb2" event={"ID":"bff71746-5b69-4358-be30-72ec1d839204","Type":"ContainerStarted","Data":"0c49780be854334d79c11f70ed9959f6612304fc424da5254670231c56b38b3d"}
Sep 16 18:40:46.164518 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:46.164477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-49ccb2" event={"ID":"bff71746-5b69-4358-be30-72ec1d839204","Type":"ContainerStarted","Data":"3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f"}
Sep 16 18:40:46.191750 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:46.191694    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7648-49ccb2" podStartSLOduration=0.993213707 podStartE2EDuration="3.191679494s" totalImagesPullingTime="2.198465787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:46.191282302 +0000 UTC m=+788.900521695" watchObservedRunningTime="2026-09-16 18:40:46.191679494 +0000 UTC m=+788.900918918"
Sep 16 18:40:46.994923 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:46.994836    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7648-49ccb2"]
Sep 16 18:40:47.008302 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:47.008269    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7648-49ccb2"]
Sep 16 18:40:47.170051 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:40:47.170020    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7648-49ccb2" container="test-curl-7648-49ccb2"
Sep 16 18:40:48.174173 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:48.174128    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7648-49ccb2" podUID="bff71746-5b69-4358-be30-72ec1d839204" containerName="test-curl-7648-49ccb2" containerID="cri-o://3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f" gracePeriod=2
Sep 16 18:40:48.921543 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:48.921497    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"]
Sep 16 18:40:48.921971 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:48.921953    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bff71746-5b69-4358-be30-72ec1d839204" containerName="test-curl-7648-49ccb2"
Sep 16 18:40:48.922033 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:48.921973    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bff71746-5b69-4358-be30-72ec1d839204" containerName="test-curl-7648-49ccb2"
Sep 16 18:40:48.922085 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:48.922079    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bff71746-5b69-4358-be30-72ec1d839204" containerName="test-curl-7648-49ccb2"
Sep 16 18:40:48.925686 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:48.925663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:40:48.928633 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:48.928608    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-4dda98\""
Sep 16 18:40:48.940420 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:48.940394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"]
Sep 16 18:40:49.001465 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.001426    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"]
Sep 16 18:40:49.004781 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.004764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:40:49.007977 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.007954    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-4dda98-dockercfg-7ckdt\""
Sep 16 18:40:49.012889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.012861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67vck\" (UniqueName: \"kubernetes.io/projected/190586c8-8189-494e-b6c6-3d5491c15d56-kube-api-access-67vck\") pod \"payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm\" (UID: \"190586c8-8189-494e-b6c6-3d5491c15d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:40:49.013085 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.013033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190586c8-8189-494e-b6c6-3d5491c15d56-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm\" (UID: \"190586c8-8189-494e-b6c6-3d5491c15d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:40:49.046868 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.046837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"]
Sep 16 18:40:49.114475 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.114443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67vck\" (UniqueName: \"kubernetes.io/projected/190586c8-8189-494e-b6c6-3d5491c15d56-kube-api-access-67vck\") pod \"payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm\" (UID: \"190586c8-8189-494e-b6c6-3d5491c15d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:40:49.114651 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.114505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4knl5\" (UniqueName: \"kubernetes.io/projected/fd91ebd5-4e35-4339-93c9-f4eedd8214db-kube-api-access-4knl5\") pod \"payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx\" (UID: \"fd91ebd5-4e35-4339-93c9-f4eedd8214db\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:40:49.114651 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.114579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190586c8-8189-494e-b6c6-3d5491c15d56-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm\" (UID: \"190586c8-8189-494e-b6c6-3d5491c15d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:40:49.114651 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.114619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd91ebd5-4e35-4339-93c9-f4eedd8214db-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx\" (UID: \"fd91ebd5-4e35-4339-93c9-f4eedd8214db\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:40:49.115273 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.115248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190586c8-8189-494e-b6c6-3d5491c15d56-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm\" (UID: \"190586c8-8189-494e-b6c6-3d5491c15d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:40:49.147714 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.147681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67vck\" (UniqueName: \"kubernetes.io/projected/190586c8-8189-494e-b6c6-3d5491c15d56-kube-api-access-67vck\") pod \"payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm\" (UID: \"190586c8-8189-494e-b6c6-3d5491c15d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:40:49.216052 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.215966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd91ebd5-4e35-4339-93c9-f4eedd8214db-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx\" (UID: \"fd91ebd5-4e35-4339-93c9-f4eedd8214db\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:40:49.216430 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.216062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4knl5\" (UniqueName: \"kubernetes.io/projected/fd91ebd5-4e35-4339-93c9-f4eedd8214db-kube-api-access-4knl5\") pod \"payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx\" (UID: \"fd91ebd5-4e35-4339-93c9-f4eedd8214db\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:40:49.216734 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.216703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd91ebd5-4e35-4339-93c9-f4eedd8214db-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx\" (UID: \"fd91ebd5-4e35-4339-93c9-f4eedd8214db\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:40:49.237544 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.237509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4knl5\" (UniqueName: \"kubernetes.io/projected/fd91ebd5-4e35-4339-93c9-f4eedd8214db-kube-api-access-4knl5\") pod \"payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx\" (UID: \"fd91ebd5-4e35-4339-93c9-f4eedd8214db\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:40:49.237724 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.237707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:40:49.316782 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.316492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:40:49.532454 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.532417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"]
Sep 16 18:40:49.534714 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:40:49.534677    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190586c8_8189_494e_b6c6_3d5491c15d56.slice/crio-8300cafc4e93f3595cebaa990ab351c2c0c1f5f3a68d20c0754ad7bee8749b74 WatchSource:0}: Error finding container 8300cafc4e93f3595cebaa990ab351c2c0c1f5f3a68d20c0754ad7bee8749b74: Status 404 returned error can't find the container with id 8300cafc4e93f3595cebaa990ab351c2c0c1f5f3a68d20c0754ad7bee8749b74
Sep 16 18:40:49.610786 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:49.610763    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"]
Sep 16 18:40:49.612923 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:40:49.612894    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd91ebd5_4e35_4339_93c9_f4eedd8214db.slice/crio-4bf185f9967dfb206965e4e376f860427bcc5cc85381aef1b36eb0ea086165ab WatchSource:0}: Error finding container 4bf185f9967dfb206965e4e376f860427bcc5cc85381aef1b36eb0ea086165ab: Status 404 returned error can't find the container with id 4bf185f9967dfb206965e4e376f860427bcc5cc85381aef1b36eb0ea086165ab
Sep 16 18:40:50.188548 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:50.188511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx" event={"ID":"fd91ebd5-4e35-4339-93c9-f4eedd8214db","Type":"ContainerStarted","Data":"4bf185f9967dfb206965e4e376f860427bcc5cc85381aef1b36eb0ea086165ab"}
Sep 16 18:40:50.190673 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:50.190636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm" event={"ID":"190586c8-8189-494e-b6c6-3d5491c15d56","Type":"ContainerStarted","Data":"8300cafc4e93f3595cebaa990ab351c2c0c1f5f3a68d20c0754ad7bee8749b74"}
Sep 16 18:40:50.387346 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:50.387294    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-49ccb2"
Sep 16 18:40:50.429037 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:50.428943    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f5tk\" (UniqueName: \"kubernetes.io/projected/bff71746-5b69-4358-be30-72ec1d839204-kube-api-access-6f5tk\") pod \"bff71746-5b69-4358-be30-72ec1d839204\" (UID: \"bff71746-5b69-4358-be30-72ec1d839204\") "
Sep 16 18:40:50.431247 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:50.431219    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff71746-5b69-4358-be30-72ec1d839204-kube-api-access-6f5tk" (OuterVolumeSpecName: "kube-api-access-6f5tk") pod "bff71746-5b69-4358-be30-72ec1d839204" (UID: "bff71746-5b69-4358-be30-72ec1d839204"). InnerVolumeSpecName "kube-api-access-6f5tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:40:50.529842 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:50.529799    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f5tk\" (UniqueName: \"kubernetes.io/projected/bff71746-5b69-4358-be30-72ec1d839204-kube-api-access-6f5tk\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:40:51.198642 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.198557    2580 generic.go:356] "Generic (PLEG): container finished" podID="bff71746-5b69-4358-be30-72ec1d839204" containerID="3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f" exitCode=137
Sep 16 18:40:51.198642 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.198607    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-49ccb2"
Sep 16 18:40:51.198861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.198675    2580 scope.go:117] "RemoveContainer" containerID="3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f"
Sep 16 18:40:51.201119 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.201094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx" event={"ID":"fd91ebd5-4e35-4339-93c9-f4eedd8214db","Type":"ContainerStarted","Data":"dc465c5dc1a70d7aba73844d463d9d47c71cfb292df9d08e847aeadae9b1634b"}
Sep 16 18:40:51.201282 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.201269    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:40:51.203050 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.203025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm" event={"ID":"190586c8-8189-494e-b6c6-3d5491c15d56","Type":"ContainerStarted","Data":"ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa"}
Sep 16 18:40:51.203180 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.203162    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:40:51.208151 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.208132    2580 scope.go:117] "RemoveContainer" containerID="3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f"
Sep 16 18:40:51.208500 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:40:51.208480    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f\": container with ID starting with 3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f not found: ID does not exist" containerID="3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f"
Sep 16 18:40:51.208557 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.208511    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f"} err="failed to get container status \"3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f\": rpc error: code = NotFound desc = could not find container \"3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f\": container with ID starting with 3996d87842772378b2a7184a94ff5647135b5bb13aa4fc10958d74f71da53d4f not found: ID does not exist"
Sep 16 18:40:51.237954 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.237912    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx" podStartSLOduration=1.9761795690000001 podStartE2EDuration="3.237897737s" totalImagesPullingTime="1.261718168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:51.236215869 +0000 UTC m=+793.945455261" watchObservedRunningTime="2026-09-16 18:40:51.237897737 +0000 UTC m=+793.947137122"
Sep 16 18:40:51.238769 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:40:51.238742    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7648-49ccb2\" is forbidden: User \"system:node:ip-10-0-141-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-16.ec2.internal' and this object" podUID="bff71746-5b69-4358-be30-72ec1d839204" pod="opendatahub/test-curl-7648-49ccb2"
Sep 16 18:40:51.278695 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.278606    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm" podStartSLOduration=1.940796505 podStartE2EDuration="3.27858228s" totalImagesPullingTime="1.337785775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:51.2772667 +0000 UTC m=+793.986506093" watchObservedRunningTime="2026-09-16 18:40:51.27858228 +0000 UTC m=+793.987821675"
Sep 16 18:40:51.783249 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:51.783214    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bff71746-5b69-4358-be30-72ec1d839204" path="/var/lib/kubelet/pods/bff71746-5b69-4358-be30-72ec1d839204/volumes"
Sep 16 18:40:57.302319 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.302279    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"]
Sep 16 18:40:57.306880 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.306860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:40:57.309477 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.309454    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-90ea7d\""
Sep 16 18:40:57.309583 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.309459    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-90ea7d-dockercfg-s4w82\""
Sep 16 18:40:57.328615 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.328588    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"]
Sep 16 18:40:57.392625 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.392595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdgt5\" (UniqueName: \"kubernetes.io/projected/f8ab76a8-a0e0-4316-bf84-b94ea1824836-kube-api-access-vdgt5\") pod \"payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6\" (UID: \"f8ab76a8-a0e0-4316-bf84-b94ea1824836\") " pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:40:57.392776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.392664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ab76a8-a0e0-4316-bf84-b94ea1824836-plugins-config-volume\") pod \"payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6\" (UID: \"f8ab76a8-a0e0-4316-bf84-b94ea1824836\") " pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:40:57.493633 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.493599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdgt5\" (UniqueName: \"kubernetes.io/projected/f8ab76a8-a0e0-4316-bf84-b94ea1824836-kube-api-access-vdgt5\") pod \"payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6\" (UID: \"f8ab76a8-a0e0-4316-bf84-b94ea1824836\") " pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:40:57.493826 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.493776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ab76a8-a0e0-4316-bf84-b94ea1824836-plugins-config-volume\") pod \"payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6\" (UID: \"f8ab76a8-a0e0-4316-bf84-b94ea1824836\") " pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:40:57.494468 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.494440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ab76a8-a0e0-4316-bf84-b94ea1824836-plugins-config-volume\") pod \"payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6\" (UID: \"f8ab76a8-a0e0-4316-bf84-b94ea1824836\") " pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:40:57.504915 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.504875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdgt5\" (UniqueName: \"kubernetes.io/projected/f8ab76a8-a0e0-4316-bf84-b94ea1824836-kube-api-access-vdgt5\") pod \"payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6\" (UID: \"f8ab76a8-a0e0-4316-bf84-b94ea1824836\") " pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:40:57.616986 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.616903    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:40:57.907211 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:57.907186    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"]
Sep 16 18:40:57.909782 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:40:57.909751    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ab76a8_a0e0_4316_bf84_b94ea1824836.slice/crio-821cd27a751c1a5cdad07601b09753c8e7add7aafc8ef0139ccffbc28a3937f8 WatchSource:0}: Error finding container 821cd27a751c1a5cdad07601b09753c8e7add7aafc8ef0139ccffbc28a3937f8: Status 404 returned error can't find the container with id 821cd27a751c1a5cdad07601b09753c8e7add7aafc8ef0139ccffbc28a3937f8
Sep 16 18:40:58.164118 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:58.164041    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:40:58.238550 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:58.238513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6" event={"ID":"f8ab76a8-a0e0-4316-bf84-b94ea1824836","Type":"ContainerStarted","Data":"26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97"}
Sep 16 18:40:58.238726 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:58.238567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6" event={"ID":"f8ab76a8-a0e0-4316-bf84-b94ea1824836","Type":"ContainerStarted","Data":"821cd27a751c1a5cdad07601b09753c8e7add7aafc8ef0139ccffbc28a3937f8"}
Sep 16 18:40:58.238726 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:58.238623    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:40:58.287758 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:40:58.287711    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6" podStartSLOduration=1.28769181 podStartE2EDuration="1.28769181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:58.286254448 +0000 UTC m=+800.995493841" watchObservedRunningTime="2026-09-16 18:40:58.28769181 +0000 UTC m=+800.996931201"
Sep 16 18:41:02.212928 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.212837    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:41:02.213376 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.213032    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:41:02.575043 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.575008    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"]
Sep 16 18:41:02.578396 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.578374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:02.581090 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.580956    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b7e002c7\""
Sep 16 18:41:02.581090 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.580956    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b7e002c7-dockercfg-g29fz\""
Sep 16 18:41:02.592399 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.592375    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"]
Sep 16 18:41:02.643425 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.643401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f4c409-75f8-44cc-ad82-d51c1750d416-plugins-config-volume\") pod \"payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x\" (UID: \"82f4c409-75f8-44cc-ad82-d51c1750d416\") " pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:02.643537 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.643466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqdxn\" (UniqueName: \"kubernetes.io/projected/82f4c409-75f8-44cc-ad82-d51c1750d416-kube-api-access-rqdxn\") pod \"payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x\" (UID: \"82f4c409-75f8-44cc-ad82-d51c1750d416\") " pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:02.674819 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.674786    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"]
Sep 16 18:41:02.678396 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.678361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:02.689497 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.689473    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"]
Sep 16 18:41:02.744458 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.744434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqdxn\" (UniqueName: \"kubernetes.io/projected/82f4c409-75f8-44cc-ad82-d51c1750d416-kube-api-access-rqdxn\") pod \"payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x\" (UID: \"82f4c409-75f8-44cc-ad82-d51c1750d416\") " pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:02.744572 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.744473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd\" (UID: \"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:02.744572 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.744519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm4gn\" (UniqueName: \"kubernetes.io/projected/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-kube-api-access-cm4gn\") pod \"payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd\" (UID: \"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:02.744572 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.744546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f4c409-75f8-44cc-ad82-d51c1750d416-plugins-config-volume\") pod \"payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x\" (UID: \"82f4c409-75f8-44cc-ad82-d51c1750d416\") " pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:02.745210 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.745191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f4c409-75f8-44cc-ad82-d51c1750d416-plugins-config-volume\") pod \"payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x\" (UID: \"82f4c409-75f8-44cc-ad82-d51c1750d416\") " pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:02.754672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.754652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqdxn\" (UniqueName: \"kubernetes.io/projected/82f4c409-75f8-44cc-ad82-d51c1750d416-kube-api-access-rqdxn\") pod \"payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x\" (UID: \"82f4c409-75f8-44cc-ad82-d51c1750d416\") " pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:02.845232 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.845172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd\" (UID: \"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:02.845232 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.845220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4gn\" (UniqueName: \"kubernetes.io/projected/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-kube-api-access-cm4gn\") pod \"payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd\" (UID: \"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:02.845878 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.845854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd\" (UID: \"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:02.857873 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.857849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4gn\" (UniqueName: \"kubernetes.io/projected/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-kube-api-access-cm4gn\") pod \"payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd\" (UID: \"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:02.888150 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.888130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:02.987720 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:02.987685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:03.172702 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:03.172676    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"]
Sep 16 18:41:03.174716 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:41:03.174687    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f4c409_75f8_44cc_ad82_d51c1750d416.slice/crio-a15ca08418cdc6199ee26fce0120c88a58a4fddebd95c5beabd249b77d63023d WatchSource:0}: Error finding container a15ca08418cdc6199ee26fce0120c88a58a4fddebd95c5beabd249b77d63023d: Status 404 returned error can't find the container with id a15ca08418cdc6199ee26fce0120c88a58a4fddebd95c5beabd249b77d63023d
Sep 16 18:41:03.264750 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:03.264710    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x" event={"ID":"82f4c409-75f8-44cc-ad82-d51c1750d416","Type":"ContainerStarted","Data":"d1d4f8b4bb69eeeb16487d06046bf4810734a9dca62673fecc1a40d1fd5873b2"}
Sep 16 18:41:03.265093 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:03.264773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x" event={"ID":"82f4c409-75f8-44cc-ad82-d51c1750d416","Type":"ContainerStarted","Data":"a15ca08418cdc6199ee26fce0120c88a58a4fddebd95c5beabd249b77d63023d"}
Sep 16 18:41:03.265093 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:03.264929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:03.275094 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:03.275048    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"]
Sep 16 18:41:03.277406 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:41:03.277379    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0bda25_1ee0_402f_b2f2_00ad9a2ddcc2.slice/crio-ac8d3f106dcc8a1e7496c5a8dd093791cf1dc648f2ec2ae26affb320cb6ca19e WatchSource:0}: Error finding container ac8d3f106dcc8a1e7496c5a8dd093791cf1dc648f2ec2ae26affb320cb6ca19e: Status 404 returned error can't find the container with id ac8d3f106dcc8a1e7496c5a8dd093791cf1dc648f2ec2ae26affb320cb6ca19e
Sep 16 18:41:03.290305 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:03.290238    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x" podStartSLOduration=1.29021932 podStartE2EDuration="1.29021932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:03.289453543 +0000 UTC m=+805.998692936" watchObservedRunningTime="2026-09-16 18:41:03.29021932 +0000 UTC m=+805.999458713"
Sep 16 18:41:04.273035 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:04.272997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd" event={"ID":"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2","Type":"ContainerStarted","Data":"6beaa32982fb019be78eb0d444d8575bd067d00437a78cdfed1dc2d1f3978de2"}
Sep 16 18:41:04.273541 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:04.273075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd" event={"ID":"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2","Type":"ContainerStarted","Data":"ac8d3f106dcc8a1e7496c5a8dd093791cf1dc648f2ec2ae26affb320cb6ca19e"}
Sep 16 18:41:04.273541 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:04.273254    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:09.246500 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:09.246469    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:41:09.291070 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:09.291021    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd" podStartSLOduration=7.291004818 podStartE2EDuration="7.291004818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:04.318827042 +0000 UTC m=+807.028066437" watchObservedRunningTime="2026-09-16 18:41:09.291004818 +0000 UTC m=+812.000244213"
Sep 16 18:41:13.387941 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:13.387909    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:41:14.275243 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:14.275208    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:15.280395 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:15.280365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:16.791481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:16.791444    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"]
Sep 16 18:41:16.794934 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:16.794916    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"
Sep 16 18:41:16.797679 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:16.797649    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:41:16.798202 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:16.798182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:41:16.811255 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:16.811231    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"]
Sep 16 18:41:16.981646 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:16.981608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4\" (UID: \"2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"
Sep 16 18:41:17.082346 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:17.082245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4\" (UID: \"2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"
Sep 16 18:41:17.082859 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:17.082839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4\" (UID: \"2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"
Sep 16 18:41:17.106231 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:17.106196    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"
Sep 16 18:41:17.397218 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:17.397190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"]
Sep 16 18:41:17.398316 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:41:17.398287    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb9d1bf_da27_4336_8bc4_1a74b0f27dd7.slice/crio-e0e6200cc2da19703cf348a33858a04bf96fcaf88366971cda3ba136464a6760 WatchSource:0}: Error finding container e0e6200cc2da19703cf348a33858a04bf96fcaf88366971cda3ba136464a6760: Status 404 returned error can't find the container with id e0e6200cc2da19703cf348a33858a04bf96fcaf88366971cda3ba136464a6760
Sep 16 18:41:18.341639 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:18.341605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4" event={"ID":"2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7","Type":"ContainerStarted","Data":"e0e6200cc2da19703cf348a33858a04bf96fcaf88366971cda3ba136464a6760"}
Sep 16 18:41:20.353681 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:20.353641    2580 generic.go:356] "Generic (PLEG): container finished" podID="2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7" containerID="fd3c06726d42aaebb084b53ea147bcb3999508938c84668d49b08030a228b7a4" exitCode=0
Sep 16 18:41:20.354031 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:20.353723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4" event={"ID":"2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7","Type":"ContainerDied","Data":"fd3c06726d42aaebb084b53ea147bcb3999508938c84668d49b08030a228b7a4"}
Sep 16 18:41:21.490008 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:21.489982    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"
Sep 16 18:41:21.520931 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:21.520900    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7-maas-api-service-ca\") pod \"2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7\" (UID: \"2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7\") "
Sep 16 18:41:21.521265 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:21.521239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7" (UID: "2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:21.622146 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:21.622076    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:41:22.365527 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:22.365495    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"
Sep 16 18:41:22.365708 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:22.365536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4" event={"ID":"2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7","Type":"ContainerDied","Data":"e0e6200cc2da19703cf348a33858a04bf96fcaf88366971cda3ba136464a6760"}
Sep 16 18:41:22.365708 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:22.365592    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0e6200cc2da19703cf348a33858a04bf96fcaf88366971cda3ba136464a6760"
Sep 16 18:41:28.144407 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.144373    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"]
Sep 16 18:41:28.144746 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.144588    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x" podUID="82f4c409-75f8-44cc-ad82-d51c1750d416" containerName="payload-processing" containerID="cri-o://d1d4f8b4bb69eeeb16487d06046bf4810734a9dca62673fecc1a40d1fd5873b2" gracePeriod=30
Sep 16 18:41:28.175216 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.175188    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"]
Sep 16 18:41:28.175423 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.175399    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd" podUID="3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2" containerName="payload-pre-processing" containerID="cri-o://6beaa32982fb019be78eb0d444d8575bd067d00437a78cdfed1dc2d1f3978de2" gracePeriod=30
Sep 16 18:41:28.397228 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.397143    2580 generic.go:356] "Generic (PLEG): container finished" podID="82f4c409-75f8-44cc-ad82-d51c1750d416" containerID="d1d4f8b4bb69eeeb16487d06046bf4810734a9dca62673fecc1a40d1fd5873b2" exitCode=0
Sep 16 18:41:28.397391 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.397218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x" event={"ID":"82f4c409-75f8-44cc-ad82-d51c1750d416","Type":"ContainerDied","Data":"d1d4f8b4bb69eeeb16487d06046bf4810734a9dca62673fecc1a40d1fd5873b2"}
Sep 16 18:41:28.398990 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.398963    2580 generic.go:356] "Generic (PLEG): container finished" podID="3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2" containerID="6beaa32982fb019be78eb0d444d8575bd067d00437a78cdfed1dc2d1f3978de2" exitCode=0
Sep 16 18:41:28.399100 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.398987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd" event={"ID":"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2","Type":"ContainerDied","Data":"6beaa32982fb019be78eb0d444d8575bd067d00437a78cdfed1dc2d1f3978de2"}
Sep 16 18:41:28.447129 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.447110    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:28.449938 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.449921    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:28.485039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.485019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f4c409-75f8-44cc-ad82-d51c1750d416-plugins-config-volume\") pod \"82f4c409-75f8-44cc-ad82-d51c1750d416\" (UID: \"82f4c409-75f8-44cc-ad82-d51c1750d416\") "
Sep 16 18:41:28.485142 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.485089    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqdxn\" (UniqueName: \"kubernetes.io/projected/82f4c409-75f8-44cc-ad82-d51c1750d416-kube-api-access-rqdxn\") pod \"82f4c409-75f8-44cc-ad82-d51c1750d416\" (UID: \"82f4c409-75f8-44cc-ad82-d51c1750d416\") "
Sep 16 18:41:28.485198 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.485141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-plugins-config-volume\") pod \"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2\" (UID: \"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2\") "
Sep 16 18:41:28.485198 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.485155    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm4gn\" (UniqueName: \"kubernetes.io/projected/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-kube-api-access-cm4gn\") pod \"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2\" (UID: \"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2\") "
Sep 16 18:41:28.485447 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.485422    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f4c409-75f8-44cc-ad82-d51c1750d416-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82f4c409-75f8-44cc-ad82-d51c1750d416" (UID: "82f4c409-75f8-44cc-ad82-d51c1750d416"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:28.485553 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.485532    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2" (UID: "3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:28.487246 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.487226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f4c409-75f8-44cc-ad82-d51c1750d416-kube-api-access-rqdxn" (OuterVolumeSpecName: "kube-api-access-rqdxn") pod "82f4c409-75f8-44cc-ad82-d51c1750d416" (UID: "82f4c409-75f8-44cc-ad82-d51c1750d416"). InnerVolumeSpecName "kube-api-access-rqdxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:28.487246 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.487234    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-kube-api-access-cm4gn" (OuterVolumeSpecName: "kube-api-access-cm4gn") pod "3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2" (UID: "3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2"). InnerVolumeSpecName "kube-api-access-cm4gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:28.586359 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.586305    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqdxn\" (UniqueName: \"kubernetes.io/projected/82f4c409-75f8-44cc-ad82-d51c1750d416-kube-api-access-rqdxn\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:41:28.586359 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.586346    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:41:28.586359 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.586361    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm4gn\" (UniqueName: \"kubernetes.io/projected/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2-kube-api-access-cm4gn\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:41:28.586612 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.586375    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f4c409-75f8-44cc-ad82-d51c1750d416-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:41:28.763879 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.763792    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"]
Sep 16 18:41:28.775410 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:28.775374    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7e002c7-kqnv4"]
Sep 16 18:41:29.404794 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.404759    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"
Sep 16 18:41:29.405247 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.404790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd" event={"ID":"3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2","Type":"ContainerDied","Data":"ac8d3f106dcc8a1e7496c5a8dd093791cf1dc648f2ec2ae26affb320cb6ca19e"}
Sep 16 18:41:29.405247 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.404859    2580 scope.go:117] "RemoveContainer" containerID="6beaa32982fb019be78eb0d444d8575bd067d00437a78cdfed1dc2d1f3978de2"
Sep 16 18:41:29.406285 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.406130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x" event={"ID":"82f4c409-75f8-44cc-ad82-d51c1750d416","Type":"ContainerDied","Data":"a15ca08418cdc6199ee26fce0120c88a58a4fddebd95c5beabd249b77d63023d"}
Sep 16 18:41:29.406285 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.406167    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"
Sep 16 18:41:29.414124 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.414088    2580 scope.go:117] "RemoveContainer" containerID="d1d4f8b4bb69eeeb16487d06046bf4810734a9dca62673fecc1a40d1fd5873b2"
Sep 16 18:41:29.506983 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.506949    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"]
Sep 16 18:41:29.562255 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.562217    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b7e002c7-54b75d6db6-4tcqd"]
Sep 16 18:41:29.649641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.649610    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"]
Sep 16 18:41:29.666099 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.666034    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b7e002c7-5b776688cf-6jc7x"]
Sep 16 18:41:29.783512 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.783480    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7" path="/var/lib/kubelet/pods/2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7/volumes"
Sep 16 18:41:29.783864 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.783850    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2" path="/var/lib/kubelet/pods/3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2/volumes"
Sep 16 18:41:29.784194 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:29.784181    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82f4c409-75f8-44cc-ad82-d51c1750d416" path="/var/lib/kubelet/pods/82f4c409-75f8-44cc-ad82-d51c1750d416/volumes"
Sep 16 18:41:30.429265 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.429227    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"]
Sep 16 18:41:30.429803 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.429782    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f4c409-75f8-44cc-ad82-d51c1750d416" containerName="payload-processing"
Sep 16 18:41:30.429881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.429806    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f4c409-75f8-44cc-ad82-d51c1750d416" containerName="payload-processing"
Sep 16 18:41:30.429881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.429820    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7" containerName="revoke-keys"
Sep 16 18:41:30.429881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.429828    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7" containerName="revoke-keys"
Sep 16 18:41:30.429881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.429854    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2" containerName="payload-pre-processing"
Sep 16 18:41:30.429881 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.429863    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2" containerName="payload-pre-processing"
Sep 16 18:41:30.430124 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.429966    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f4c409-75f8-44cc-ad82-d51c1750d416" containerName="payload-processing"
Sep 16 18:41:30.430124 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.429980    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb9d1bf-da27-4336-8bc4-1a74b0f27dd7" containerName="revoke-keys"
Sep 16 18:41:30.430124 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.429989    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b0bda25-1ee0-402f-b2f2-00ad9a2ddcc2" containerName="payload-pre-processing"
Sep 16 18:41:30.434118 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.434097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.438961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.438941    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:41:30.439084 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.438963    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b88c0110-gw-openshift-default-dockercfg-qg242\""
Sep 16 18:41:30.468310 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.468281    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"]
Sep 16 18:41:30.504552 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.504524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-podinfo\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.504698 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.504567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061f9924-615f-476e-9bd7-2cb1fc0649ef-istiod-ca-cert\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.504698 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.504592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-certs\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.504698 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.504640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-credential-socket\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.504698 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.504675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-envoy\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.504833 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.504698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bl4\" (UniqueName: \"kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-kube-api-access-56bl4\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.504833 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.504730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-socket\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.504833 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.504756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-data\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.504833 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.504788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-token\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.605990 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.605957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-data\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606195 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-token\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606195 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-podinfo\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606195 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061f9924-615f-476e-9bd7-2cb1fc0649ef-istiod-ca-cert\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606195 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-certs\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606450 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-credential-socket\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606450 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-envoy\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606450 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bl4\" (UniqueName: \"kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-kube-api-access-56bl4\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606450 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-socket\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606650 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-data\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606650 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-certs\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606739 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-credential-socket\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606785 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-socket\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.606883 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.606862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061f9924-615f-476e-9bd7-2cb1fc0649ef-istiod-ca-cert\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.608526 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.608506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-envoy\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.608667 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.608651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-podinfo\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.638193 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.638164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-token\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.638312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.638264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bl4\" (UniqueName: \"kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-kube-api-access-56bl4\") pod \"e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") " pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:30.744860 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:30.744776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:31.094279 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:31.094253    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"]
Sep 16 18:41:31.096722 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:41:31.096695    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061f9924_615f_476e_9bd7_2cb1fc0649ef.slice/crio-f28be64d6274a94e40d7ec8cace8f494d4d244d938f0ac7ff3c5c73a547fb1ae WatchSource:0}: Error finding container f28be64d6274a94e40d7ec8cace8f494d4d244d938f0ac7ff3c5c73a547fb1ae: Status 404 returned error can't find the container with id f28be64d6274a94e40d7ec8cace8f494d4d244d938f0ac7ff3c5c73a547fb1ae
Sep 16 18:41:31.418576 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:31.418484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" event={"ID":"061f9924-615f-476e-9bd7-2cb1fc0649ef","Type":"ContainerStarted","Data":"f28be64d6274a94e40d7ec8cace8f494d4d244d938f0ac7ff3c5c73a547fb1ae"}
Sep 16 18:41:33.036231 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:33.036193    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:41:33.433811 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:33.433767    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:41:33.433918 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:33.433846    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:41:33.433918 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:33.433875    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:41:34.435379 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:34.435343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" event={"ID":"061f9924-615f-476e-9bd7-2cb1fc0649ef","Type":"ContainerStarted","Data":"b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849"}
Sep 16 18:41:34.498574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:34.498520    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" podStartSLOduration=2.163812674 podStartE2EDuration="4.49850554s" totalImagesPullingTime="2.334692866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:34.497168333 +0000 UTC m=+837.206407725" watchObservedRunningTime="2026-09-16 18:41:34.49850554 +0000 UTC m=+837.207745006"
Sep 16 18:41:34.745223 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:34.745131    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:34.750694 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:34.750667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:35.440080 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:35.440048    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:35.440879 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:35.440861    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:41:46.292169 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.292132    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"]
Sep 16 18:41:46.303112 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.303085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:41:46.307928 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.307906    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b88c0110-dockercfg-6k2xv\""
Sep 16 18:41:46.308070 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.307929    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b88c0110\""
Sep 16 18:41:46.335895 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.335873    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"]
Sep 16 18:41:46.392716 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.392685    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"]
Sep 16 18:41:46.395914 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.395899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:41:46.441630 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.441605    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"]
Sep 16 18:41:46.455561 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.455526    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2\" (UID: \"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2\") " pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:41:46.455699 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.455590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppng4\" (UniqueName: \"kubernetes.io/projected/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-kube-api-access-ppng4\") pod \"payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2\" (UID: \"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2\") " pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:41:46.556383 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.556283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7fmg\" (UniqueName: \"kubernetes.io/projected/ef86e668-5412-4ed2-a7a8-2a02077b25d7-kube-api-access-x7fmg\") pod \"payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt\" (UID: \"ef86e668-5412-4ed2-a7a8-2a02077b25d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:41:46.556383 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.556350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppng4\" (UniqueName: \"kubernetes.io/projected/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-kube-api-access-ppng4\") pod \"payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2\" (UID: \"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2\") " pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:41:46.556563 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.556408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef86e668-5412-4ed2-a7a8-2a02077b25d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt\" (UID: \"ef86e668-5412-4ed2-a7a8-2a02077b25d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:41:46.556563 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.556499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2\" (UID: \"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2\") " pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:41:46.557173 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.557155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2\" (UID: \"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2\") " pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:41:46.573687 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.573656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppng4\" (UniqueName: \"kubernetes.io/projected/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-kube-api-access-ppng4\") pod \"payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2\" (UID: \"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2\") " pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:41:46.612422 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.612389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:41:46.657722 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.657682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef86e668-5412-4ed2-a7a8-2a02077b25d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt\" (UID: \"ef86e668-5412-4ed2-a7a8-2a02077b25d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:41:46.657869 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.657771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7fmg\" (UniqueName: \"kubernetes.io/projected/ef86e668-5412-4ed2-a7a8-2a02077b25d7-kube-api-access-x7fmg\") pod \"payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt\" (UID: \"ef86e668-5412-4ed2-a7a8-2a02077b25d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:41:46.658521 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.658502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef86e668-5412-4ed2-a7a8-2a02077b25d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt\" (UID: \"ef86e668-5412-4ed2-a7a8-2a02077b25d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:41:46.689165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.689134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7fmg\" (UniqueName: \"kubernetes.io/projected/ef86e668-5412-4ed2-a7a8-2a02077b25d7-kube-api-access-x7fmg\") pod \"payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt\" (UID: \"ef86e668-5412-4ed2-a7a8-2a02077b25d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:41:46.705919 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.705896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:41:46.902282 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:46.902190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"]
Sep 16 18:41:46.906829 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:41:46.906801    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffd3f36_ff0a_449f_9dc9_19cb7fce75b2.slice/crio-45a87ae2d0c717a5dee13f0a4f40dd9783b9e615993b6c13fac15d125bc7ce96 WatchSource:0}: Error finding container 45a87ae2d0c717a5dee13f0a4f40dd9783b9e615993b6c13fac15d125bc7ce96: Status 404 returned error can't find the container with id 45a87ae2d0c717a5dee13f0a4f40dd9783b9e615993b6c13fac15d125bc7ce96
Sep 16 18:41:47.031228 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:47.030931    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"]
Sep 16 18:41:47.034248 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:41:47.034215    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef86e668_5412_4ed2_a7a8_2a02077b25d7.slice/crio-2828dfafeee611c4fee9a7e7e490d56285da8902459e0a7c493e74975f0544cc WatchSource:0}: Error finding container 2828dfafeee611c4fee9a7e7e490d56285da8902459e0a7c493e74975f0544cc: Status 404 returned error can't find the container with id 2828dfafeee611c4fee9a7e7e490d56285da8902459e0a7c493e74975f0544cc
Sep 16 18:41:47.499799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:47.499762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt" event={"ID":"ef86e668-5412-4ed2-a7a8-2a02077b25d7","Type":"ContainerStarted","Data":"a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9"}
Sep 16 18:41:47.500280 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:47.499826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt" event={"ID":"ef86e668-5412-4ed2-a7a8-2a02077b25d7","Type":"ContainerStarted","Data":"2828dfafeee611c4fee9a7e7e490d56285da8902459e0a7c493e74975f0544cc"}
Sep 16 18:41:47.500280 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:47.499856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:41:47.501941 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:47.501906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2" event={"ID":"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2","Type":"ContainerStarted","Data":"6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f"}
Sep 16 18:41:47.502074 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:47.501963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2" event={"ID":"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2","Type":"ContainerStarted","Data":"45a87ae2d0c717a5dee13f0a4f40dd9783b9e615993b6c13fac15d125bc7ce96"}
Sep 16 18:41:47.502074 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:47.502029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:41:47.554214 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:47.554161    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt" podStartSLOduration=1.554145182 podStartE2EDuration="1.554145182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:47.542161358 +0000 UTC m=+850.251400751" watchObservedRunningTime="2026-09-16 18:41:47.554145182 +0000 UTC m=+850.263384573"
Sep 16 18:41:47.598584 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:47.598523    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2" podStartSLOduration=1.598505922 podStartE2EDuration="1.598505922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:47.597801159 +0000 UTC m=+850.307040553" watchObservedRunningTime="2026-09-16 18:41:47.598505922 +0000 UTC m=+850.307745314"
Sep 16 18:41:51.437811 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:51.437781    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:41:55.687214 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.687167    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"]
Sep 16 18:41:55.692418 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.692394    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.696619 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.696601    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-662432\"/\"default-dockercfg-ntgt5\""
Sep 16 18:41:55.696851 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.696833    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-662432\"/\"openshift-service-ca.crt\""
Sep 16 18:41:55.697861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.697838    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-662432\"/\"kube-root-ca.crt\""
Sep 16 18:41:55.697969 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.697870    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-662432\"/\"simulator-w4-662432-kserve-self-signed-certs\""
Sep 16 18:41:55.716393 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.716362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"]
Sep 16 18:41:55.848660 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.848621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-dshm\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.848831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.848679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-model-cache\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.848831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.848744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55c2cfb-9598-4621-adc5-a059a975a236-tls-certs\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.848831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.848780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmnrs\" (UniqueName: \"kubernetes.io/projected/e55c2cfb-9598-4621-adc5-a059a975a236-kube-api-access-gmnrs\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.848963 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.848863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-tmp-dir\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.848963 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.848891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-home\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.949627 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.949543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-tmp-dir\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.949627 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.949578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-home\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.949627 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.949627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-dshm\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.949899 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.949659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-model-cache\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.949899 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.949687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55c2cfb-9598-4621-adc5-a059a975a236-tls-certs\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.949899 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.949705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmnrs\" (UniqueName: \"kubernetes.io/projected/e55c2cfb-9598-4621-adc5-a059a975a236-kube-api-access-gmnrs\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.950055 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.949949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-tmp-dir\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.950055 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.950026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-home\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.950138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.950118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-model-cache\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.951896 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.951876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-dshm\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.952203 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.952186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55c2cfb-9598-4621-adc5-a059a975a236-tls-certs\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:55.968406 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:55.968378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmnrs\" (UniqueName: \"kubernetes.io/projected/e55c2cfb-9598-4621-adc5-a059a975a236-kube-api-access-gmnrs\") pod \"simulator-w4-662432-kserve-c7b445cb8-7zw4q\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") " pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:56.004097 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:56.004062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:41:56.307319 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:56.307235    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"]
Sep 16 18:41:56.310174 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:41:56.310146    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode55c2cfb_9598_4621_adc5_a059a975a236.slice/crio-7c6ff53bdb6e9ebad03211347ac3a9a808cae38886d5e0ded4e862ac73927c4d WatchSource:0}: Error finding container 7c6ff53bdb6e9ebad03211347ac3a9a808cae38886d5e0ded4e862ac73927c4d: Status 404 returned error can't find the container with id 7c6ff53bdb6e9ebad03211347ac3a9a808cae38886d5e0ded4e862ac73927c4d
Sep 16 18:41:56.547142 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:56.547103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q" event={"ID":"e55c2cfb-9598-4621-adc5-a059a975a236","Type":"ContainerStarted","Data":"53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc"}
Sep 16 18:41:56.547301 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:56.547153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q" event={"ID":"e55c2cfb-9598-4621-adc5-a059a975a236","Type":"ContainerStarted","Data":"7c6ff53bdb6e9ebad03211347ac3a9a808cae38886d5e0ded4e862ac73927c4d"}
Sep 16 18:41:56.588515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:56.588407    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q" podStartSLOduration=1.588390066 podStartE2EDuration="1.588390066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:56.585108922 +0000 UTC m=+859.294348318" watchObservedRunningTime="2026-09-16 18:41:56.588390066 +0000 UTC m=+859.297629457"
Sep 16 18:41:58.509922 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:58.509891    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:41:58.510397 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:41:58.510358    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:42:01.187529 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:01.187489    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"]
Sep 16 18:42:01.189905 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:01.189888    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"
Sep 16 18:42:01.193116 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:01.193097    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:42:01.193369 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:01.193354    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:42:01.210918 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:01.210888    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"]
Sep 16 18:42:01.303095 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:01.303063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43286d3e-2e26-4047-b73d-fa4d1a95b8b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm\" (UID: \"43286d3e-2e26-4047-b73d-fa4d1a95b8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"
Sep 16 18:42:01.403966 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:01.403930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43286d3e-2e26-4047-b73d-fa4d1a95b8b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm\" (UID: \"43286d3e-2e26-4047-b73d-fa4d1a95b8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"
Sep 16 18:42:01.404618 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:01.404599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43286d3e-2e26-4047-b73d-fa4d1a95b8b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm\" (UID: \"43286d3e-2e26-4047-b73d-fa4d1a95b8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"
Sep 16 18:42:01.501139 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:01.501049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"
Sep 16 18:42:01.809495 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:01.809470    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"]
Sep 16 18:42:01.812325 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:42:01.812296    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43286d3e_2e26_4047_b73d_fa4d1a95b8b9.slice/crio-04fb999cc1a3fab59104af1eec6b9f8358e45a8c2252465ccb7c6fe8f75250b9 WatchSource:0}: Error finding container 04fb999cc1a3fab59104af1eec6b9f8358e45a8c2252465ccb7c6fe8f75250b9: Status 404 returned error can't find the container with id 04fb999cc1a3fab59104af1eec6b9f8358e45a8c2252465ccb7c6fe8f75250b9
Sep 16 18:42:02.594411 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:02.594381    2580 generic.go:356] "Generic (PLEG): container finished" podID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerID="b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849" exitCode=0
Sep 16 18:42:02.594816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:02.594458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" event={"ID":"061f9924-615f-476e-9bd7-2cb1fc0649ef","Type":"ContainerDied","Data":"b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849"}
Sep 16 18:42:02.594948 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:02.594929    2580 scope.go:117] "RemoveContainer" containerID="b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849"
Sep 16 18:42:02.596093 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:02.596061    2580 generic.go:356] "Generic (PLEG): container finished" podID="43286d3e-2e26-4047-b73d-fa4d1a95b8b9" containerID="cbc12192dee367a94d68282ab9e47dcd3939cc2487de4438ed99e02a4189e306" exitCode=0
Sep 16 18:42:02.596216 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:02.596128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm" event={"ID":"43286d3e-2e26-4047-b73d-fa4d1a95b8b9","Type":"ContainerDied","Data":"cbc12192dee367a94d68282ab9e47dcd3939cc2487de4438ed99e02a4189e306"}
Sep 16 18:42:02.596216 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:02.596160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm" event={"ID":"43286d3e-2e26-4047-b73d-fa4d1a95b8b9","Type":"ContainerStarted","Data":"04fb999cc1a3fab59104af1eec6b9f8358e45a8c2252465ccb7c6fe8f75250b9"}
Sep 16 18:42:02.596302 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:02.596246    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:02.596353 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:02.596312    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:02.596395 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:02.596359    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:03.608170 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:03.606254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" event={"ID":"061f9924-615f-476e-9bd7-2cb1fc0649ef","Type":"ContainerStarted","Data":"f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e"}
Sep 16 18:42:03.745359 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:03.745316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:42:03.746412 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:03.746384    2580 patch_prober.go:28] interesting pod/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 18:42:03.746516 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:03.746441    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 18:42:03.758463 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:03.758444    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"
Sep 16 18:42:03.929407 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:03.929280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43286d3e-2e26-4047-b73d-fa4d1a95b8b9-maas-api-service-ca\") pod \"43286d3e-2e26-4047-b73d-fa4d1a95b8b9\" (UID: \"43286d3e-2e26-4047-b73d-fa4d1a95b8b9\") "
Sep 16 18:42:03.929651 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:03.929627    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43286d3e-2e26-4047-b73d-fa4d1a95b8b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43286d3e-2e26-4047-b73d-fa4d1a95b8b9" (UID: "43286d3e-2e26-4047-b73d-fa4d1a95b8b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:04.030411 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:04.030381    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43286d3e-2e26-4047-b73d-fa4d1a95b8b9-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:04.612875 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:04.612844    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"
Sep 16 18:42:04.613242 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:04.612882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm" event={"ID":"43286d3e-2e26-4047-b73d-fa4d1a95b8b9","Type":"ContainerDied","Data":"04fb999cc1a3fab59104af1eec6b9f8358e45a8c2252465ccb7c6fe8f75250b9"}
Sep 16 18:42:04.613242 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:04.612930    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04fb999cc1a3fab59104af1eec6b9f8358e45a8c2252465ccb7c6fe8f75250b9"
Sep 16 18:42:04.745235 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:04.745198    2580 patch_prober.go:28] interesting pod/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 18:42:04.745413 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:04.745247    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 18:42:05.440584 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:05.440550    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:42:05.746046 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:05.745956    2580 patch_prober.go:28] interesting pod/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 18:42:05.746046 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:05.746014    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 18:42:06.005171 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:06.005083    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:42:06.005171 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:06.005129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:42:06.017977 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:06.017954    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:42:06.638166 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:06.638133    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:42:06.745156 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:06.745125    2580 patch_prober.go:28] interesting pod/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 18:42:06.745309 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:06.745180    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 18:42:07.745498 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:07.745455    2580 patch_prober.go:28] interesting pod/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 18:42:07.745970 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:07.745521    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 18:42:08.745525 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:08.745490    2580 patch_prober.go:28] interesting pod/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 18:42:08.745908 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:08.745550    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 18:42:09.749625 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:09.749594    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:42:09.750759 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:09.750737    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:42:12.529323 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:12.529281    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:42:13.672759 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:13.672725    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"]
Sep 16 18:42:13.673230 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:13.672971    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2" podUID="cffd3f36-ff0a-449f-9dc9-19cb7fce75b2" containerName="payload-processing" containerID="cri-o://6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f" gracePeriod=30
Sep 16 18:42:13.678753 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:13.678723    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"]
Sep 16 18:42:13.679003 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:13.678958    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt" podUID="ef86e668-5412-4ed2-a7a8-2a02077b25d7" containerName="payload-pre-processing" containerID="cri-o://a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9" gracePeriod=30
Sep 16 18:42:13.964649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:13.964625    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:42:13.968059 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:13.968025    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:42:13.971532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:13.971511    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"]
Sep 16 18:42:13.976726 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:13.976703    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b88c0110-nzctm"]
Sep 16 18:42:14.013632 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.013600    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-plugins-config-volume\") pod \"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2\" (UID: \"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2\") "
Sep 16 18:42:14.013823 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.013711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7fmg\" (UniqueName: \"kubernetes.io/projected/ef86e668-5412-4ed2-a7a8-2a02077b25d7-kube-api-access-x7fmg\") pod \"ef86e668-5412-4ed2-a7a8-2a02077b25d7\" (UID: \"ef86e668-5412-4ed2-a7a8-2a02077b25d7\") "
Sep 16 18:42:14.013823 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.013798    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppng4\" (UniqueName: \"kubernetes.io/projected/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-kube-api-access-ppng4\") pod \"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2\" (UID: \"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2\") "
Sep 16 18:42:14.013955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.013827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef86e668-5412-4ed2-a7a8-2a02077b25d7-plugins-config-volume\") pod \"ef86e668-5412-4ed2-a7a8-2a02077b25d7\" (UID: \"ef86e668-5412-4ed2-a7a8-2a02077b25d7\") "
Sep 16 18:42:14.014139 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.014073    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cffd3f36-ff0a-449f-9dc9-19cb7fce75b2" (UID: "cffd3f36-ff0a-449f-9dc9-19cb7fce75b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:14.014459 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.014434    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef86e668-5412-4ed2-a7a8-2a02077b25d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef86e668-5412-4ed2-a7a8-2a02077b25d7" (UID: "ef86e668-5412-4ed2-a7a8-2a02077b25d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:14.016025 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.015999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef86e668-5412-4ed2-a7a8-2a02077b25d7-kube-api-access-x7fmg" (OuterVolumeSpecName: "kube-api-access-x7fmg") pod "ef86e668-5412-4ed2-a7a8-2a02077b25d7" (UID: "ef86e668-5412-4ed2-a7a8-2a02077b25d7"). InnerVolumeSpecName "kube-api-access-x7fmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:14.016603 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.016582    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-kube-api-access-ppng4" (OuterVolumeSpecName: "kube-api-access-ppng4") pod "cffd3f36-ff0a-449f-9dc9-19cb7fce75b2" (UID: "cffd3f36-ff0a-449f-9dc9-19cb7fce75b2"). InnerVolumeSpecName "kube-api-access-ppng4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:14.114831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.114788    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7fmg\" (UniqueName: \"kubernetes.io/projected/ef86e668-5412-4ed2-a7a8-2a02077b25d7-kube-api-access-x7fmg\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:14.114831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.114828    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppng4\" (UniqueName: \"kubernetes.io/projected/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-kube-api-access-ppng4\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:14.115088 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.114846    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef86e668-5412-4ed2-a7a8-2a02077b25d7-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:14.115088 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.114861    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:14.124429 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.124394    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"]
Sep 16 18:42:14.124626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.124603    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy" containerID="cri-o://f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e" gracePeriod=30
Sep 16 18:42:14.665776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.665738    2580 generic.go:356] "Generic (PLEG): container finished" podID="ef86e668-5412-4ed2-a7a8-2a02077b25d7" containerID="a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9" exitCode=0
Sep 16 18:42:14.665776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.665781    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"
Sep 16 18:42:14.666039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.665811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt" event={"ID":"ef86e668-5412-4ed2-a7a8-2a02077b25d7","Type":"ContainerDied","Data":"a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9"}
Sep 16 18:42:14.666039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.665862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt" event={"ID":"ef86e668-5412-4ed2-a7a8-2a02077b25d7","Type":"ContainerDied","Data":"2828dfafeee611c4fee9a7e7e490d56285da8902459e0a7c493e74975f0544cc"}
Sep 16 18:42:14.666039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.665885    2580 scope.go:117] "RemoveContainer" containerID="a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9"
Sep 16 18:42:14.667404 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.667376    2580 generic.go:356] "Generic (PLEG): container finished" podID="cffd3f36-ff0a-449f-9dc9-19cb7fce75b2" containerID="6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f" exitCode=0
Sep 16 18:42:14.667550 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.667419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"
Sep 16 18:42:14.667550 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.667447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2" event={"ID":"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2","Type":"ContainerDied","Data":"6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f"}
Sep 16 18:42:14.667550 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.667491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2" event={"ID":"cffd3f36-ff0a-449f-9dc9-19cb7fce75b2","Type":"ContainerDied","Data":"45a87ae2d0c717a5dee13f0a4f40dd9783b9e615993b6c13fac15d125bc7ce96"}
Sep 16 18:42:14.675905 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.675659    2580 scope.go:117] "RemoveContainer" containerID="a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9"
Sep 16 18:42:14.676229 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:42:14.675987    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9\": container with ID starting with a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9 not found: ID does not exist" containerID="a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9"
Sep 16 18:42:14.676229 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.676027    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9"} err="failed to get container status \"a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9\": rpc error: code = NotFound desc = could not find container \"a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9\": container with ID starting with a0c48758bea28f3c4a1a46e567d015e864db92fc590080eb00fa4004b0c484e9 not found: ID does not exist"
Sep 16 18:42:14.676229 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.676058    2580 scope.go:117] "RemoveContainer" containerID="6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f"
Sep 16 18:42:14.685151 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.685130    2580 scope.go:117] "RemoveContainer" containerID="6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f"
Sep 16 18:42:14.685449 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:42:14.685423    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f\": container with ID starting with 6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f not found: ID does not exist" containerID="6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f"
Sep 16 18:42:14.685541 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.685459    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f"} err="failed to get container status \"6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f\": rpc error: code = NotFound desc = could not find container \"6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f\": container with ID starting with 6791433ed555d31fb5eb23bb6045b23071dd73f1a8f69f7f77f5f88861be248f not found: ID does not exist"
Sep 16 18:42:14.696567 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.696545    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"]
Sep 16 18:42:14.700144 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.700124    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b88c0110-556d549bd6-w5kr2"]
Sep 16 18:42:14.718985 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.718957    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"]
Sep 16 18:42:14.727074 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.727054    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b88c0110-68c5d57bd6-vm4wt"]
Sep 16 18:42:14.903114 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.903073    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"]
Sep 16 18:42:14.903459 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.903444    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef86e668-5412-4ed2-a7a8-2a02077b25d7" containerName="payload-pre-processing"
Sep 16 18:42:14.903459 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.903458    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef86e668-5412-4ed2-a7a8-2a02077b25d7" containerName="payload-pre-processing"
Sep 16 18:42:14.903649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.903467    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43286d3e-2e26-4047-b73d-fa4d1a95b8b9" containerName="revoke-keys"
Sep 16 18:42:14.903649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.903473    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="43286d3e-2e26-4047-b73d-fa4d1a95b8b9" containerName="revoke-keys"
Sep 16 18:42:14.903649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.903487    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffd3f36-ff0a-449f-9dc9-19cb7fce75b2" containerName="payload-processing"
Sep 16 18:42:14.903649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.903493    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffd3f36-ff0a-449f-9dc9-19cb7fce75b2" containerName="payload-processing"
Sep 16 18:42:14.903649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.903561    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cffd3f36-ff0a-449f-9dc9-19cb7fce75b2" containerName="payload-processing"
Sep 16 18:42:14.903649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.903570    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef86e668-5412-4ed2-a7a8-2a02077b25d7" containerName="payload-pre-processing"
Sep 16 18:42:14.903649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.903578    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="43286d3e-2e26-4047-b73d-fa4d1a95b8b9" containerName="revoke-keys"
Sep 16 18:42:14.906355 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.906315    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:14.908737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.908717    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c6d0650c-openshift-default-dockercfg-4zgk4\""
Sep 16 18:42:14.922084 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.922017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-data\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:14.922185 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.922086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-certs\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:14.922185 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.922165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-socket\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:14.922302 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.922192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-credential-socket\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:14.922302 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.922220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-token\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:14.922302 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.922265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-envoy\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:14.922302 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.922289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istiod-ca-cert\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:14.922502 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.922359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-podinfo\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:14.922502 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.922402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltlvt\" (UniqueName: \"kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-kube-api-access-ltlvt\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:14.923855 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:14.923834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"]
Sep 16 18:42:15.023424 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.023386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltlvt\" (UniqueName: \"kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-kube-api-access-ltlvt\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.023610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.023471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-data\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.023610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.023536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-certs\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.023610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.023606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-socket\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.023772 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.023630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-credential-socket\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.023772 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.023659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-token\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.023772 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.023689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-envoy\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.023772 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.023712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istiod-ca-cert\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.023772 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.023749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-podinfo\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.024092 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.023905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-certs\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.024092 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.024054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-data\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.024384 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.024362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-socket\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.024589 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.024397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-credential-socket\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.024769 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.024745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istiod-ca-cert\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.026414 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.026395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-envoy\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.026771 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.026751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-podinfo\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.034736 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.034713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-token\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.035308 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.035285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltlvt\" (UniqueName: \"kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-kube-api-access-ltlvt\") pod \"e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") " pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.218365 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.218261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:15.500465 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.500386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"]
Sep 16 18:42:15.503270 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:42:15.503243    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ddf1a4_a5b1_456b_bf42_f6fb4350ca89.slice/crio-e66bd55b1f17bf40848e9da8cc6c478c3d70cb94dd1e24e12bd8aebf02c507f4 WatchSource:0}: Error finding container e66bd55b1f17bf40848e9da8cc6c478c3d70cb94dd1e24e12bd8aebf02c507f4: Status 404 returned error can't find the container with id e66bd55b1f17bf40848e9da8cc6c478c3d70cb94dd1e24e12bd8aebf02c507f4
Sep 16 18:42:15.505648 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.505618    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:15.505723 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.505680    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:15.505723 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.505709    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:15.678081 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.678039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" event={"ID":"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89","Type":"ContainerStarted","Data":"383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418"}
Sep 16 18:42:15.678564 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.678145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" event={"ID":"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89","Type":"ContainerStarted","Data":"e66bd55b1f17bf40848e9da8cc6c478c3d70cb94dd1e24e12bd8aebf02c507f4"}
Sep 16 18:42:15.710246 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.710194    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" podStartSLOduration=1.710177834 podStartE2EDuration="1.710177834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:15.708211966 +0000 UTC m=+878.417451358" watchObservedRunningTime="2026-09-16 18:42:15.710177834 +0000 UTC m=+878.419417270"
Sep 16 18:42:15.783680 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.783643    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43286d3e-2e26-4047-b73d-fa4d1a95b8b9" path="/var/lib/kubelet/pods/43286d3e-2e26-4047-b73d-fa4d1a95b8b9/volumes"
Sep 16 18:42:15.784024 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.784008    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cffd3f36-ff0a-449f-9dc9-19cb7fce75b2" path="/var/lib/kubelet/pods/cffd3f36-ff0a-449f-9dc9-19cb7fce75b2/volumes"
Sep 16 18:42:15.784390 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:15.784373    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef86e668-5412-4ed2-a7a8-2a02077b25d7" path="/var/lib/kubelet/pods/ef86e668-5412-4ed2-a7a8-2a02077b25d7/volumes"
Sep 16 18:42:16.218629 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:16.218540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:16.223774 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:16.223747    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:16.682753 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:16.682724    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:16.683735 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:16.683713    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:19.377755 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.377729    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:42:19.471906 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.471806    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-socket\") pod \"061f9924-615f-476e-9bd7-2cb1fc0649ef\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") "
Sep 16 18:42:19.471906 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.471841    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-data\") pod \"061f9924-615f-476e-9bd7-2cb1fc0649ef\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") "
Sep 16 18:42:19.471906 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.471868    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-envoy\") pod \"061f9924-615f-476e-9bd7-2cb1fc0649ef\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") "
Sep 16 18:42:19.472208 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.471931    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-certs\") pod \"061f9924-615f-476e-9bd7-2cb1fc0649ef\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") "
Sep 16 18:42:19.472208 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.471953    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-token\") pod \"061f9924-615f-476e-9bd7-2cb1fc0649ef\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") "
Sep 16 18:42:19.472208 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.471972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56bl4\" (UniqueName: \"kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-kube-api-access-56bl4\") pod \"061f9924-615f-476e-9bd7-2cb1fc0649ef\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") "
Sep 16 18:42:19.472208 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.472013    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061f9924-615f-476e-9bd7-2cb1fc0649ef-istiod-ca-cert\") pod \"061f9924-615f-476e-9bd7-2cb1fc0649ef\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") "
Sep 16 18:42:19.472208 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.472051    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-podinfo\") pod \"061f9924-615f-476e-9bd7-2cb1fc0649ef\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") "
Sep 16 18:42:19.472208 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.472085    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-credential-socket\") pod \"061f9924-615f-476e-9bd7-2cb1fc0649ef\" (UID: \"061f9924-615f-476e-9bd7-2cb1fc0649ef\") "
Sep 16 18:42:19.472208 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.472172    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "061f9924-615f-476e-9bd7-2cb1fc0649ef" (UID: "061f9924-615f-476e-9bd7-2cb1fc0649ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:19.472686 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.472242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "061f9924-615f-476e-9bd7-2cb1fc0649ef" (UID: "061f9924-615f-476e-9bd7-2cb1fc0649ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:19.472686 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.472370    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "061f9924-615f-476e-9bd7-2cb1fc0649ef" (UID: "061f9924-615f-476e-9bd7-2cb1fc0649ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:19.472686 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.472439    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "061f9924-615f-476e-9bd7-2cb1fc0649ef" (UID: "061f9924-615f-476e-9bd7-2cb1fc0649ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:19.472686 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.472463    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:19.472686 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.472484    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-data\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:19.472686 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.472661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061f9924-615f-476e-9bd7-2cb1fc0649ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "061f9924-615f-476e-9bd7-2cb1fc0649ef" (UID: "061f9924-615f-476e-9bd7-2cb1fc0649ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:19.474407 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.474377    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-kube-api-access-56bl4" (OuterVolumeSpecName: "kube-api-access-56bl4") pod "061f9924-615f-476e-9bd7-2cb1fc0649ef" (UID: "061f9924-615f-476e-9bd7-2cb1fc0649ef"). InnerVolumeSpecName "kube-api-access-56bl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:19.474529 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.474470    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "061f9924-615f-476e-9bd7-2cb1fc0649ef" (UID: "061f9924-615f-476e-9bd7-2cb1fc0649ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:19.474674 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.474655    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "061f9924-615f-476e-9bd7-2cb1fc0649ef" (UID: "061f9924-615f-476e-9bd7-2cb1fc0649ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:42:19.474780 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.474766    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "061f9924-615f-476e-9bd7-2cb1fc0649ef" (UID: "061f9924-615f-476e-9bd7-2cb1fc0649ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:19.573674 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.573642    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-envoy\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:19.573674 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.573667    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-workload-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:19.573674 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.573676    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-token\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:19.573878 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.573684    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56bl4\" (UniqueName: \"kubernetes.io/projected/061f9924-615f-476e-9bd7-2cb1fc0649ef-kube-api-access-56bl4\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:19.573878 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.573692    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061f9924-615f-476e-9bd7-2cb1fc0649ef-istiod-ca-cert\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:19.573878 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.573701    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061f9924-615f-476e-9bd7-2cb1fc0649ef-istio-podinfo\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:19.573878 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.573708    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061f9924-615f-476e-9bd7-2cb1fc0649ef-credential-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:19.705239 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.705202    2580 generic.go:356] "Generic (PLEG): container finished" podID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerID="f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e" exitCode=0
Sep 16 18:42:19.705449 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.705260    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"
Sep 16 18:42:19.705449 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.705277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" event={"ID":"061f9924-615f-476e-9bd7-2cb1fc0649ef","Type":"ContainerDied","Data":"f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e"}
Sep 16 18:42:19.705449 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.705397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6" event={"ID":"061f9924-615f-476e-9bd7-2cb1fc0649ef","Type":"ContainerDied","Data":"f28be64d6274a94e40d7ec8cace8f494d4d244d938f0ac7ff3c5c73a547fb1ae"}
Sep 16 18:42:19.705449 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.705435    2580 scope.go:117] "RemoveContainer" containerID="f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e"
Sep 16 18:42:19.716613 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.716596    2580 scope.go:117] "RemoveContainer" containerID="b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849"
Sep 16 18:42:19.724693 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.724675    2580 scope.go:117] "RemoveContainer" containerID="f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e"
Sep 16 18:42:19.724942 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:42:19.724922    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e\": container with ID starting with f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e not found: ID does not exist" containerID="f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e"
Sep 16 18:42:19.725006 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.724949    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e"} err="failed to get container status \"f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e\": rpc error: code = NotFound desc = could not find container \"f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e\": container with ID starting with f8a1b75d3ca9c900ace41233468f78e79a0084f78fe38ccd5b26fff94c0a323e not found: ID does not exist"
Sep 16 18:42:19.725006 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.724964    2580 scope.go:117] "RemoveContainer" containerID="b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849"
Sep 16 18:42:19.725181 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:42:19.725166    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849\": container with ID starting with b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849 not found: ID does not exist" containerID="b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849"
Sep 16 18:42:19.725221 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.725185    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849"} err="failed to get container status \"b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849\": rpc error: code = NotFound desc = could not find container \"b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849\": container with ID starting with b742f351153ccd7de3962ed349ee7d4df16496287ff7278c413b4e86dfe24849 not found: ID does not exist"
Sep 16 18:42:19.733599 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.733575    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"]
Sep 16 18:42:19.750167 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.750147    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b88c0110-gw-openshift-default-9d8dc7cf6-9pkn6"]
Sep 16 18:42:19.782681 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:19.782661    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" path="/var/lib/kubelet/pods/061f9924-615f-476e-9bd7-2cb1fc0649ef/volumes"
Sep 16 18:42:25.408687 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.408611    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"]
Sep 16 18:42:25.409079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.408960    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy"
Sep 16 18:42:25.409079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.408971    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy"
Sep 16 18:42:25.409079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.408982    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy"
Sep 16 18:42:25.409079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.408988    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy"
Sep 16 18:42:25.409079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.409046    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy"
Sep 16 18:42:25.409079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.409057    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="061f9924-615f-476e-9bd7-2cb1fc0649ef" containerName="istio-proxy"
Sep 16 18:42:25.412207 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.412191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:25.416347 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.416306    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c6d0650c\""
Sep 16 18:42:25.416486 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.416362    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c6d0650c-dockercfg-psrnc\""
Sep 16 18:42:25.427304 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.427277    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"]
Sep 16 18:42:25.509762 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.509737    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"]
Sep 16 18:42:25.511953 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.511937    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:25.531087 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.531065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"]
Sep 16 18:42:25.532905 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.532877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580883c8-c535-4a4f-9f47-33493ee8a30f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc\" (UID: \"580883c8-c535-4a4f-9f47-33493ee8a30f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:25.533005 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.532985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8x8v\" (UniqueName: \"kubernetes.io/projected/580883c8-c535-4a4f-9f47-33493ee8a30f-kube-api-access-z8x8v\") pod \"payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc\" (UID: \"580883c8-c535-4a4f-9f47-33493ee8a30f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:25.634303 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.634267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqv6\" (UniqueName: \"kubernetes.io/projected/4b2f5f58-0a73-4e49-b410-38f0e946923c-kube-api-access-vsqv6\") pod \"payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw\" (UID: \"4b2f5f58-0a73-4e49-b410-38f0e946923c\") " pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:25.634483 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.634365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580883c8-c535-4a4f-9f47-33493ee8a30f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc\" (UID: \"580883c8-c535-4a4f-9f47-33493ee8a30f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:25.634483 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.634415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2f5f58-0a73-4e49-b410-38f0e946923c-plugins-config-volume\") pod \"payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw\" (UID: \"4b2f5f58-0a73-4e49-b410-38f0e946923c\") " pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:25.634483 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.634457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8x8v\" (UniqueName: \"kubernetes.io/projected/580883c8-c535-4a4f-9f47-33493ee8a30f-kube-api-access-z8x8v\") pod \"payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc\" (UID: \"580883c8-c535-4a4f-9f47-33493ee8a30f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:25.635031 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.635010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580883c8-c535-4a4f-9f47-33493ee8a30f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc\" (UID: \"580883c8-c535-4a4f-9f47-33493ee8a30f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:25.647069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.647043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8x8v\" (UniqueName: \"kubernetes.io/projected/580883c8-c535-4a4f-9f47-33493ee8a30f-kube-api-access-z8x8v\") pod \"payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc\" (UID: \"580883c8-c535-4a4f-9f47-33493ee8a30f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:25.735689 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.735612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2f5f58-0a73-4e49-b410-38f0e946923c-plugins-config-volume\") pod \"payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw\" (UID: \"4b2f5f58-0a73-4e49-b410-38f0e946923c\") " pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:25.735835 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.735683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqv6\" (UniqueName: \"kubernetes.io/projected/4b2f5f58-0a73-4e49-b410-38f0e946923c-kube-api-access-vsqv6\") pod \"payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw\" (UID: \"4b2f5f58-0a73-4e49-b410-38f0e946923c\") " pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:25.736298 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.736277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2f5f58-0a73-4e49-b410-38f0e946923c-plugins-config-volume\") pod \"payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw\" (UID: \"4b2f5f58-0a73-4e49-b410-38f0e946923c\") " pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:25.746429 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.746409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqv6\" (UniqueName: \"kubernetes.io/projected/4b2f5f58-0a73-4e49-b410-38f0e946923c-kube-api-access-vsqv6\") pod \"payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw\" (UID: \"4b2f5f58-0a73-4e49-b410-38f0e946923c\") " pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:25.751226 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.751211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:25.821709 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:25.821644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:26.030497 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:26.030469    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"]
Sep 16 18:42:26.032435 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:42:26.032391    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580883c8_c535_4a4f_9f47_33493ee8a30f.slice/crio-200fa474b8e5034958f70f82d3298e760e4fc9863ab65d887005509b1f2da754 WatchSource:0}: Error finding container 200fa474b8e5034958f70f82d3298e760e4fc9863ab65d887005509b1f2da754: Status 404 returned error can't find the container with id 200fa474b8e5034958f70f82d3298e760e4fc9863ab65d887005509b1f2da754
Sep 16 18:42:26.103146 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:26.103123    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"]
Sep 16 18:42:26.104743 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:42:26.104717    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2f5f58_0a73_4e49_b410_38f0e946923c.slice/crio-74dc52b04c279210a7b35cd045b090fbdc569e792a998ba8955632329ab61254 WatchSource:0}: Error finding container 74dc52b04c279210a7b35cd045b090fbdc569e792a998ba8955632329ab61254: Status 404 returned error can't find the container with id 74dc52b04c279210a7b35cd045b090fbdc569e792a998ba8955632329ab61254
Sep 16 18:42:26.742193 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:26.742161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw" event={"ID":"4b2f5f58-0a73-4e49-b410-38f0e946923c","Type":"ContainerStarted","Data":"90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a"}
Sep 16 18:42:26.742685 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:26.742223    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:26.742685 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:26.742243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw" event={"ID":"4b2f5f58-0a73-4e49-b410-38f0e946923c","Type":"ContainerStarted","Data":"74dc52b04c279210a7b35cd045b090fbdc569e792a998ba8955632329ab61254"}
Sep 16 18:42:26.743620 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:26.743593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc" event={"ID":"580883c8-c535-4a4f-9f47-33493ee8a30f","Type":"ContainerStarted","Data":"db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4"}
Sep 16 18:42:26.743699 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:26.743647    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:26.743699 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:26.743666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc" event={"ID":"580883c8-c535-4a4f-9f47-33493ee8a30f","Type":"ContainerStarted","Data":"200fa474b8e5034958f70f82d3298e760e4fc9863ab65d887005509b1f2da754"}
Sep 16 18:42:26.775085 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:26.775027    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw" podStartSLOduration=1.77501064 podStartE2EDuration="1.77501064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:26.773364032 +0000 UTC m=+889.482603425" watchObservedRunningTime="2026-09-16 18:42:26.77501064 +0000 UTC m=+889.484250060"
Sep 16 18:42:26.796971 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:26.796906    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc" podStartSLOduration=1.79688903 podStartE2EDuration="1.79688903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:26.795412436 +0000 UTC m=+889.504651858" watchObservedRunningTime="2026-09-16 18:42:26.79688903 +0000 UTC m=+889.506128422"
Sep 16 18:42:32.330613 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:32.330578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:42:37.750544 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:37.750493    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:37.750951 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:37.750700    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:37.751287 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:37.751262    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:42:37.751585 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:37.751564    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:42:42.839951 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:42.839918    2580 generic.go:356] "Generic (PLEG): container finished" podID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerID="383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418" exitCode=0
Sep 16 18:42:42.840394 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:42.839960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" event={"ID":"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89","Type":"ContainerDied","Data":"383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418"}
Sep 16 18:42:42.840449 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:42.840415    2580 scope.go:117] "RemoveContainer" containerID="383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418"
Sep 16 18:42:42.841579 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:42.841553    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:42.841661 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:42.841648    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:42.841720 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:42.841709    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:43.102688 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.102603    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"]
Sep 16 18:42:43.105590 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.105563    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"
Sep 16 18:42:43.108158 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.108140    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:42:43.108272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.108249    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:42:43.126689 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.126653    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"]
Sep 16 18:42:43.203983 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.203952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d737f0-fd23-40b4-a1ea-04271cbd1434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6d0650c-xffft\" (UID: \"c5d737f0-fd23-40b4-a1ea-04271cbd1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"
Sep 16 18:42:43.304908 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.304874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d737f0-fd23-40b4-a1ea-04271cbd1434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6d0650c-xffft\" (UID: \"c5d737f0-fd23-40b4-a1ea-04271cbd1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"
Sep 16 18:42:43.305513 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.305492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d737f0-fd23-40b4-a1ea-04271cbd1434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6d0650c-xffft\" (UID: \"c5d737f0-fd23-40b4-a1ea-04271cbd1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"
Sep 16 18:42:43.419776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.419354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"
Sep 16 18:42:43.724444 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.724417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"]
Sep 16 18:42:43.726692 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:42:43.726661    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d737f0_fd23_40b4_a1ea_04271cbd1434.slice/crio-958a328dc95e01d32c7fe2b46f68ace3577aea92a63ec106ae2c04a51bbf6f43 WatchSource:0}: Error finding container 958a328dc95e01d32c7fe2b46f68ace3577aea92a63ec106ae2c04a51bbf6f43: Status 404 returned error can't find the container with id 958a328dc95e01d32c7fe2b46f68ace3577aea92a63ec106ae2c04a51bbf6f43
Sep 16 18:42:43.846639 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.846605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft" event={"ID":"c5d737f0-fd23-40b4-a1ea-04271cbd1434","Type":"ContainerStarted","Data":"958a328dc95e01d32c7fe2b46f68ace3577aea92a63ec106ae2c04a51bbf6f43"}
Sep 16 18:42:43.848404 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:43.848375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" event={"ID":"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89","Type":"ContainerStarted","Data":"2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864"}
Sep 16 18:42:44.218809 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:44.218774    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:44.219871 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:44.219846    2580 patch_prober.go:28] interesting pod/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 18:42:44.219985 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:44.219906    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 18:42:44.855321 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:44.855228    2580 generic.go:356] "Generic (PLEG): container finished" podID="c5d737f0-fd23-40b4-a1ea-04271cbd1434" containerID="b18e9777149d02d392f8bed281fd88bc9117327e762437267159360a7c1324f3" exitCode=0
Sep 16 18:42:44.855321 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:44.855290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft" event={"ID":"c5d737f0-fd23-40b4-a1ea-04271cbd1434","Type":"ContainerDied","Data":"b18e9777149d02d392f8bed281fd88bc9117327e762437267159360a7c1324f3"}
Sep 16 18:42:45.218946 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:45.218844    2580 patch_prober.go:28] interesting pod/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 18:42:45.218946 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:45.218908    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 18:42:46.027375 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:46.027352    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"
Sep 16 18:42:46.132877 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:46.132790    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d737f0-fd23-40b4-a1ea-04271cbd1434-maas-api-service-ca\") pod \"c5d737f0-fd23-40b4-a1ea-04271cbd1434\" (UID: \"c5d737f0-fd23-40b4-a1ea-04271cbd1434\") "
Sep 16 18:42:46.133141 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:46.133115    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d737f0-fd23-40b4-a1ea-04271cbd1434-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5d737f0-fd23-40b4-a1ea-04271cbd1434" (UID: "c5d737f0-fd23-40b4-a1ea-04271cbd1434"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:46.218750 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:46.218714    2580 patch_prober.go:28] interesting pod/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 18:42:46.218920 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:46.218792    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 18:42:46.233998 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:46.233971    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d737f0-fd23-40b4-a1ea-04271cbd1434-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:46.682979 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:46.682929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:46.871018 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:46.870970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft" event={"ID":"c5d737f0-fd23-40b4-a1ea-04271cbd1434","Type":"ContainerDied","Data":"958a328dc95e01d32c7fe2b46f68ace3577aea92a63ec106ae2c04a51bbf6f43"}
Sep 16 18:42:46.871237 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:46.871041    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="958a328dc95e01d32c7fe2b46f68ace3577aea92a63ec106ae2c04a51bbf6f43"
Sep 16 18:42:46.871237 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:46.871109    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"
Sep 16 18:42:47.219729 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:47.219688    2580 patch_prober.go:28] interesting pod/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 18:42:47.220121 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:47.219760    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 18:42:48.219181 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:48.219143    2580 patch_prober.go:28] interesting pod/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 18:42:48.219407 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:48.219212    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 18:42:49.223422 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:49.223391    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:49.224436 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:49.224420    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:52.967991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:52.967954    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"]
Sep 16 18:42:52.968473 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:52.968226    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw" podUID="4b2f5f58-0a73-4e49-b410-38f0e946923c" containerName="payload-processing" containerID="cri-o://90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a" gracePeriod=30
Sep 16 18:42:52.978114 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:52.977391    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"]
Sep 16 18:42:52.978114 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:52.977922    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc" podUID="580883c8-c535-4a4f-9f47-33493ee8a30f" containerName="payload-pre-processing" containerID="cri-o://db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4" gracePeriod=30
Sep 16 18:42:53.182444 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.182411    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"]
Sep 16 18:42:53.198522 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.198435    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6d0650c-xffft"]
Sep 16 18:42:53.258124 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.258097    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:53.261742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.261721    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:53.299766 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.299729    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580883c8-c535-4a4f-9f47-33493ee8a30f-plugins-config-volume\") pod \"580883c8-c535-4a4f-9f47-33493ee8a30f\" (UID: \"580883c8-c535-4a4f-9f47-33493ee8a30f\") "
Sep 16 18:42:53.300212 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.300184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580883c8-c535-4a4f-9f47-33493ee8a30f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "580883c8-c535-4a4f-9f47-33493ee8a30f" (UID: "580883c8-c535-4a4f-9f47-33493ee8a30f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:53.300348 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.300271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8x8v\" (UniqueName: \"kubernetes.io/projected/580883c8-c535-4a4f-9f47-33493ee8a30f-kube-api-access-z8x8v\") pod \"580883c8-c535-4a4f-9f47-33493ee8a30f\" (UID: \"580883c8-c535-4a4f-9f47-33493ee8a30f\") "
Sep 16 18:42:53.300435 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.300385    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2f5f58-0a73-4e49-b410-38f0e946923c-plugins-config-volume\") pod \"4b2f5f58-0a73-4e49-b410-38f0e946923c\" (UID: \"4b2f5f58-0a73-4e49-b410-38f0e946923c\") "
Sep 16 18:42:53.300503 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.300488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsqv6\" (UniqueName: \"kubernetes.io/projected/4b2f5f58-0a73-4e49-b410-38f0e946923c-kube-api-access-vsqv6\") pod \"4b2f5f58-0a73-4e49-b410-38f0e946923c\" (UID: \"4b2f5f58-0a73-4e49-b410-38f0e946923c\") "
Sep 16 18:42:53.300843 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.300822    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580883c8-c535-4a4f-9f47-33493ee8a30f-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:53.301647 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.301605    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2f5f58-0a73-4e49-b410-38f0e946923c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b2f5f58-0a73-4e49-b410-38f0e946923c" (UID: "4b2f5f58-0a73-4e49-b410-38f0e946923c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:53.304196 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.304013    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580883c8-c535-4a4f-9f47-33493ee8a30f-kube-api-access-z8x8v" (OuterVolumeSpecName: "kube-api-access-z8x8v") pod "580883c8-c535-4a4f-9f47-33493ee8a30f" (UID: "580883c8-c535-4a4f-9f47-33493ee8a30f"). InnerVolumeSpecName "kube-api-access-z8x8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:53.304551 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.304525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2f5f58-0a73-4e49-b410-38f0e946923c-kube-api-access-vsqv6" (OuterVolumeSpecName: "kube-api-access-vsqv6") pod "4b2f5f58-0a73-4e49-b410-38f0e946923c" (UID: "4b2f5f58-0a73-4e49-b410-38f0e946923c"). InnerVolumeSpecName "kube-api-access-vsqv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:53.402131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.402092    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsqv6\" (UniqueName: \"kubernetes.io/projected/4b2f5f58-0a73-4e49-b410-38f0e946923c-kube-api-access-vsqv6\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:53.402131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.402126    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8x8v\" (UniqueName: \"kubernetes.io/projected/580883c8-c535-4a4f-9f47-33493ee8a30f-kube-api-access-z8x8v\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:53.402131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.402138    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2f5f58-0a73-4e49-b410-38f0e946923c-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:53.783484 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.783439    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5d737f0-fd23-40b4-a1ea-04271cbd1434" path="/var/lib/kubelet/pods/c5d737f0-fd23-40b4-a1ea-04271cbd1434/volumes"
Sep 16 18:42:53.911126 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.911087    2580 generic.go:356] "Generic (PLEG): container finished" podID="4b2f5f58-0a73-4e49-b410-38f0e946923c" containerID="90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a" exitCode=0
Sep 16 18:42:53.911322 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.911137    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"
Sep 16 18:42:53.911322 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.911179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw" event={"ID":"4b2f5f58-0a73-4e49-b410-38f0e946923c","Type":"ContainerDied","Data":"90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a"}
Sep 16 18:42:53.911322 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.911241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw" event={"ID":"4b2f5f58-0a73-4e49-b410-38f0e946923c","Type":"ContainerDied","Data":"74dc52b04c279210a7b35cd045b090fbdc569e792a998ba8955632329ab61254"}
Sep 16 18:42:53.911322 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.911272    2580 scope.go:117] "RemoveContainer" containerID="90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a"
Sep 16 18:42:53.912610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.912586    2580 generic.go:356] "Generic (PLEG): container finished" podID="580883c8-c535-4a4f-9f47-33493ee8a30f" containerID="db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4" exitCode=0
Sep 16 18:42:53.912724 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.912625    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"
Sep 16 18:42:53.912724 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.912625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc" event={"ID":"580883c8-c535-4a4f-9f47-33493ee8a30f","Type":"ContainerDied","Data":"db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4"}
Sep 16 18:42:53.912724 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.912685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc" event={"ID":"580883c8-c535-4a4f-9f47-33493ee8a30f","Type":"ContainerDied","Data":"200fa474b8e5034958f70f82d3298e760e4fc9863ab65d887005509b1f2da754"}
Sep 16 18:42:53.920356 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.920321    2580 scope.go:117] "RemoveContainer" containerID="90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a"
Sep 16 18:42:53.920617 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:42:53.920591    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a\": container with ID starting with 90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a not found: ID does not exist" containerID="90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a"
Sep 16 18:42:53.920680 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.920619    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a"} err="failed to get container status \"90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a\": rpc error: code = NotFound desc = could not find container \"90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a\": container with ID starting with 90cd437070c37b36b1853a3bb644b1fefea3c73ce550d8ddf8bfd568bc56403a not found: ID does not exist"
Sep 16 18:42:53.920680 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.920635    2580 scope.go:117] "RemoveContainer" containerID="db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4"
Sep 16 18:42:53.928002 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.927979    2580 scope.go:117] "RemoveContainer" containerID="db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4"
Sep 16 18:42:53.928345 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:42:53.928308    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4\": container with ID starting with db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4 not found: ID does not exist" containerID="db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4"
Sep 16 18:42:53.928434 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.928357    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4"} err="failed to get container status \"db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4\": rpc error: code = NotFound desc = could not find container \"db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4\": container with ID starting with db0bb7e04410a5531e3a293088f6b1bed80c7dbf8f5537d70072ee071b05c2d4 not found: ID does not exist"
Sep 16 18:42:53.929981 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.929962    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"]
Sep 16 18:42:53.936749 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.936726    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6d0650c-7d645f844f-8tprw"]
Sep 16 18:42:53.951982 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.951950    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"]
Sep 16 18:42:53.957134 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:53.957106    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6d0650c-86cff8bbb6-vcspc"]
Sep 16 18:42:54.490214 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:54.490182    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"]
Sep 16 18:42:54.490636 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:54.490419    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy" containerID="cri-o://2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864" gracePeriod=30
Sep 16 18:42:55.261071 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.261043    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"]
Sep 16 18:42:55.261459 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.261445    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2f5f58-0a73-4e49-b410-38f0e946923c" containerName="payload-processing"
Sep 16 18:42:55.261532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.261462    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2f5f58-0a73-4e49-b410-38f0e946923c" containerName="payload-processing"
Sep 16 18:42:55.261532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.261479    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d737f0-fd23-40b4-a1ea-04271cbd1434" containerName="revoke-keys"
Sep 16 18:42:55.261532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.261484    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d737f0-fd23-40b4-a1ea-04271cbd1434" containerName="revoke-keys"
Sep 16 18:42:55.261532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.261502    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580883c8-c535-4a4f-9f47-33493ee8a30f" containerName="payload-pre-processing"
Sep 16 18:42:55.261532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.261511    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="580883c8-c535-4a4f-9f47-33493ee8a30f" containerName="payload-pre-processing"
Sep 16 18:42:55.261754 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.261566    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="580883c8-c535-4a4f-9f47-33493ee8a30f" containerName="payload-pre-processing"
Sep 16 18:42:55.261754 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.261578    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2f5f58-0a73-4e49-b410-38f0e946923c" containerName="payload-processing"
Sep 16 18:42:55.261754 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.261587    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5d737f0-fd23-40b4-a1ea-04271cbd1434" containerName="revoke-keys"
Sep 16 18:42:55.264294 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.264275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.268569 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.268546    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0c558678-openshift-default-dockercfg-74f75\""
Sep 16 18:42:55.280459 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.280433    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"]
Sep 16 18:42:55.318507 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.318483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9l9x\" (UniqueName: \"kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-kube-api-access-p9l9x\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.318647 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.318520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-credential-socket\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.318647 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.318538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40780c93-968c-4147-ac5e-2437983c56d8-istio-podinfo\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.318776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.318678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-certs\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.318776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.318713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-envoy\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.318776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.318760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-istio-token\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.318912 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.318785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-socket\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.318912 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.318815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-data\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.318912 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.318881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40780c93-968c-4147-ac5e-2437983c56d8-istiod-ca-cert\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.419616 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.419580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-data\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.419796 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.419640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40780c93-968c-4147-ac5e-2437983c56d8-istiod-ca-cert\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.419796 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.419680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9l9x\" (UniqueName: \"kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-kube-api-access-p9l9x\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.419796 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.419709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-credential-socket\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.419796 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.419724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40780c93-968c-4147-ac5e-2437983c56d8-istio-podinfo\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.420031 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.419803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-certs\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.420031 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.419823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-envoy\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.420031 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.419886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-istio-token\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.420031 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.419974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-socket\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.420268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.420048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-data\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.420268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.420122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-credential-socket\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.420268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.420237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-certs\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.420424 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.420386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-socket\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.420494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.420476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40780c93-968c-4147-ac5e-2437983c56d8-istiod-ca-cert\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.422133 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.422113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-envoy\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.422271 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.422253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40780c93-968c-4147-ac5e-2437983c56d8-istio-podinfo\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.431422 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.431395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-istio-token\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.431589 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.431571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9l9x\" (UniqueName: \"kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-kube-api-access-p9l9x\") pod \"e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") " pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.575793 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.575766    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:55.790470 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.790435    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b2f5f58-0a73-4e49-b410-38f0e946923c" path="/var/lib/kubelet/pods/4b2f5f58-0a73-4e49-b410-38f0e946923c/volumes"
Sep 16 18:42:55.790798 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.790785    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="580883c8-c535-4a4f-9f47-33493ee8a30f" path="/var/lib/kubelet/pods/580883c8-c535-4a4f-9f47-33493ee8a30f/volumes"
Sep 16 18:42:55.872864 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.872841    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"]
Sep 16 18:42:55.875142 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:42:55.875110    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40780c93_968c_4147_ac5e_2437983c56d8.slice/crio-043c4c67c90320ac006990853990521fa0049cdd5d11d7e7933ca22cd304611f WatchSource:0}: Error finding container 043c4c67c90320ac006990853990521fa0049cdd5d11d7e7933ca22cd304611f: Status 404 returned error can't find the container with id 043c4c67c90320ac006990853990521fa0049cdd5d11d7e7933ca22cd304611f
Sep 16 18:42:55.877310 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.877277    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:55.877407 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.877357    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:55.877407 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.877383    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:42:55.927802 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:55.927775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" event={"ID":"40780c93-968c-4147-ac5e-2437983c56d8","Type":"ContainerStarted","Data":"043c4c67c90320ac006990853990521fa0049cdd5d11d7e7933ca22cd304611f"}
Sep 16 18:42:56.936818 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:56.936782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" event={"ID":"40780c93-968c-4147-ac5e-2437983c56d8","Type":"ContainerStarted","Data":"fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959"}
Sep 16 18:42:56.964832 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:56.964775    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" podStartSLOduration=1.964759603 podStartE2EDuration="1.964759603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:56.962234986 +0000 UTC m=+919.671474378" watchObservedRunningTime="2026-09-16 18:42:56.964759603 +0000 UTC m=+919.673998995"
Sep 16 18:42:57.576665 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:57.576627    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:57.582041 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:57.582012    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:57.942237 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:57.942149    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:57.943301 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:57.943275    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:42:59.742201 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.742178    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:59.864661 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.864626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-certs\") pod \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") "
Sep 16 18:42:59.864849 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.864684    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-data\") pod \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") "
Sep 16 18:42:59.864849 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.864716    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltlvt\" (UniqueName: \"kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-kube-api-access-ltlvt\") pod \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") "
Sep 16 18:42:59.864849 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.864748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istiod-ca-cert\") pod \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") "
Sep 16 18:42:59.864849 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.864783    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-token\") pod \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") "
Sep 16 18:42:59.865068 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.864935    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" (UID: "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:59.865068 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.864949    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-data" (OuterVolumeSpecName: "istio-data") pod "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" (UID: "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:59.865068 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.864975    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-podinfo\") pod \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") "
Sep 16 18:42:59.865068 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.865020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-envoy\") pod \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") "
Sep 16 18:42:59.865271 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.865102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-socket\") pod \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") "
Sep 16 18:42:59.865271 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.865149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-credential-socket\") pod \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\" (UID: \"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89\") "
Sep 16 18:42:59.865415 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.865268    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" (UID: "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:59.865540 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.865519    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" (UID: "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:59.865635 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.865602    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:59.865635 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.865622    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-data\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:59.865739 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.865637    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istiod-ca-cert\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:59.865795 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.865747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" (UID: "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:59.867228 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.867201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-token" (OuterVolumeSpecName: "istio-token") pod "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" (UID: "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:59.867371 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.867302    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" (UID: "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:42:59.867486 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.867469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-kube-api-access-ltlvt" (OuterVolumeSpecName: "kube-api-access-ltlvt") pod "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" (UID: "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89"). InnerVolumeSpecName "kube-api-access-ltlvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:59.867548 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.867533    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" (UID: "91ddf1a4-a5b1-456b-bf42-f6fb4350ca89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:59.953646 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.953609    2580 generic.go:356] "Generic (PLEG): container finished" podID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerID="2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864" exitCode=0
Sep 16 18:42:59.953816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.953656    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"
Sep 16 18:42:59.953816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.953685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" event={"ID":"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89","Type":"ContainerDied","Data":"2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864"}
Sep 16 18:42:59.953816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.953749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn" event={"ID":"91ddf1a4-a5b1-456b-bf42-f6fb4350ca89","Type":"ContainerDied","Data":"e66bd55b1f17bf40848e9da8cc6c478c3d70cb94dd1e24e12bd8aebf02c507f4"}
Sep 16 18:42:59.953816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.953773    2580 scope.go:117] "RemoveContainer" containerID="2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864"
Sep 16 18:42:59.963440 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.963422    2580 scope.go:117] "RemoveContainer" containerID="383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418"
Sep 16 18:42:59.966365 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.966318    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltlvt\" (UniqueName: \"kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-kube-api-access-ltlvt\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:59.966515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.966369    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-token\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:59.966515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.966385    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-podinfo\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:59.966515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.966397    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-istio-envoy\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:59.966515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.966409    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-workload-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:59.966515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.966422    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89-credential-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:42:59.971965 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.971942    2580 scope.go:117] "RemoveContainer" containerID="2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864"
Sep 16 18:42:59.972226 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:42:59.972207    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864\": container with ID starting with 2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864 not found: ID does not exist" containerID="2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864"
Sep 16 18:42:59.972286 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.972234    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864"} err="failed to get container status \"2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864\": rpc error: code = NotFound desc = could not find container \"2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864\": container with ID starting with 2ac4a9741744fa966873ae7feeaa63eb4b3919641f67c704bc2a269ea0751864 not found: ID does not exist"
Sep 16 18:42:59.972286 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.972250    2580 scope.go:117] "RemoveContainer" containerID="383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418"
Sep 16 18:42:59.972545 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:42:59.972528    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418\": container with ID starting with 383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418 not found: ID does not exist" containerID="383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418"
Sep 16 18:42:59.972592 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.972550    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418"} err="failed to get container status \"383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418\": rpc error: code = NotFound desc = could not find container \"383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418\": container with ID starting with 383af84115207b745d709ad066745031ce29889f0248c2cd0ba0751fb01cb418 not found: ID does not exist"
Sep 16 18:42:59.985158 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.985127    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"]
Sep 16 18:42:59.991818 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:42:59.991795    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c6d0650c-openshift-default-69cdfdd6db-q5dcn"]
Sep 16 18:43:00.007205 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:00.007178    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:43:01.783059 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:01.783027    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" path="/var/lib/kubelet/pods/91ddf1a4-a5b1-456b-bf42-f6fb4350ca89/volumes"
Sep 16 18:43:03.514384 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.514345    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"]
Sep 16 18:43:03.514895 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.514875    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy"
Sep 16 18:43:03.514980 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.514899    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy"
Sep 16 18:43:03.514980 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.514911    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy"
Sep 16 18:43:03.514980 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.514920    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy"
Sep 16 18:43:03.515133 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.515010    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy"
Sep 16 18:43:03.515133 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.515023    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ddf1a4-a5b1-456b-bf42-f6fb4350ca89" containerName="istio-proxy"
Sep 16 18:43:03.517918 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.517895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.521830 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.521810    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-90ea7d\"/\"autores-model-90ea7d-kserve-self-signed-certs\""
Sep 16 18:43:03.521944 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.521846    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-90ea7d\"/\"default-dockercfg-25r6l\""
Sep 16 18:43:03.521944 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.521811    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-90ea7d\"/\"kube-root-ca.crt\""
Sep 16 18:43:03.521944 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.521811    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-90ea7d\"/\"openshift-service-ca.crt\""
Sep 16 18:43:03.545808 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.545783    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"]
Sep 16 18:43:03.601432 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.601397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/748f92c9-b51e-4266-88c8-719a0dcd5557-tls-certs\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.601571 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.601459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9vzg\" (UniqueName: \"kubernetes.io/projected/748f92c9-b51e-4266-88c8-719a0dcd5557-kube-api-access-f9vzg\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.601571 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.601486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-model-cache\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.601571 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.601523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-home\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.601683 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.601589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-dshm\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.601683 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.601640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-tmp-dir\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.702432 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.702391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/748f92c9-b51e-4266-88c8-719a0dcd5557-tls-certs\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.702610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.702499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9vzg\" (UniqueName: \"kubernetes.io/projected/748f92c9-b51e-4266-88c8-719a0dcd5557-kube-api-access-f9vzg\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.702610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.702526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-model-cache\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.702610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.702577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-home\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.702610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.702598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-dshm\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.702781 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.702678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-tmp-dir\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.702977 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.702955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-home\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.703049 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.702987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-model-cache\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.703049 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.703017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-tmp-dir\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.704695 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.704675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-dshm\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.705079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.705058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/748f92c9-b51e-4266-88c8-719a0dcd5557-tls-certs\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.712378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.712357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9vzg\" (UniqueName: \"kubernetes.io/projected/748f92c9-b51e-4266-88c8-719a0dcd5557-kube-api-access-f9vzg\") pod \"autores-model-90ea7d-kserve-7674785d7f-gff6d\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") " pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:03.826946 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:03.826913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:04.108018 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.107989    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"]
Sep 16 18:43:04.110274 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:43:04.110245    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod748f92c9_b51e_4266_88c8_719a0dcd5557.slice/crio-709a662e6d28c715c7e4d010a0796bcd5e1dd0077833944298a485c0e5b8f791 WatchSource:0}: Error finding container 709a662e6d28c715c7e4d010a0796bcd5e1dd0077833944298a485c0e5b8f791: Status 404 returned error can't find the container with id 709a662e6d28c715c7e4d010a0796bcd5e1dd0077833944298a485c0e5b8f791
Sep 16 18:43:04.197671 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.197639    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"]
Sep 16 18:43:04.200132 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.200109    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:04.202664 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.202642    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0c558678\""
Sep 16 18:43:04.202828 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.202744    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0c558678-dockercfg-qvqpn\""
Sep 16 18:43:04.213489 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.213465    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"]
Sep 16 18:43:04.257385 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.257346    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"]
Sep 16 18:43:04.259675 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.259637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:04.274761 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.274737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"]
Sep 16 18:43:04.307734 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.307708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cghr\" (UniqueName: \"kubernetes.io/projected/a049b337-43ea-4d64-b797-e00b1178d276-kube-api-access-6cghr\") pod \"payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq\" (UID: \"a049b337-43ea-4d64-b797-e00b1178d276\") " pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:04.307850 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.307744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a049b337-43ea-4d64-b797-e00b1178d276-plugins-config-volume\") pod \"payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq\" (UID: \"a049b337-43ea-4d64-b797-e00b1178d276\") " pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:04.307850 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.307770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8lrg\" (UniqueName: \"kubernetes.io/projected/60eebd82-09af-4aa9-bd33-fd9376cd731d-kube-api-access-w8lrg\") pod \"payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2\" (UID: \"60eebd82-09af-4aa9-bd33-fd9376cd731d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:04.307850 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.307847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eebd82-09af-4aa9-bd33-fd9376cd731d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2\" (UID: \"60eebd82-09af-4aa9-bd33-fd9376cd731d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:04.408887 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.408785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eebd82-09af-4aa9-bd33-fd9376cd731d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2\" (UID: \"60eebd82-09af-4aa9-bd33-fd9376cd731d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:04.408887 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.408853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cghr\" (UniqueName: \"kubernetes.io/projected/a049b337-43ea-4d64-b797-e00b1178d276-kube-api-access-6cghr\") pod \"payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq\" (UID: \"a049b337-43ea-4d64-b797-e00b1178d276\") " pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:04.408887 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.408886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a049b337-43ea-4d64-b797-e00b1178d276-plugins-config-volume\") pod \"payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq\" (UID: \"a049b337-43ea-4d64-b797-e00b1178d276\") " pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:04.409174 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.408911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8lrg\" (UniqueName: \"kubernetes.io/projected/60eebd82-09af-4aa9-bd33-fd9376cd731d-kube-api-access-w8lrg\") pod \"payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2\" (UID: \"60eebd82-09af-4aa9-bd33-fd9376cd731d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:04.409670 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.409644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eebd82-09af-4aa9-bd33-fd9376cd731d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2\" (UID: \"60eebd82-09af-4aa9-bd33-fd9376cd731d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:04.410034 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.410016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a049b337-43ea-4d64-b797-e00b1178d276-plugins-config-volume\") pod \"payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq\" (UID: \"a049b337-43ea-4d64-b797-e00b1178d276\") " pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:04.419492 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.419469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8lrg\" (UniqueName: \"kubernetes.io/projected/60eebd82-09af-4aa9-bd33-fd9376cd731d-kube-api-access-w8lrg\") pod \"payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2\" (UID: \"60eebd82-09af-4aa9-bd33-fd9376cd731d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:04.419600 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.419518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cghr\" (UniqueName: \"kubernetes.io/projected/a049b337-43ea-4d64-b797-e00b1178d276-kube-api-access-6cghr\") pod \"payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq\" (UID: \"a049b337-43ea-4d64-b797-e00b1178d276\") " pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:04.510609 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.510565    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:04.569676 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.569643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:04.786066 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.786038    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"]
Sep 16 18:43:04.787889 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:43:04.787862    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda049b337_43ea_4d64_b797_e00b1178d276.slice/crio-44ff890feb5c12c309451cf661e1f2fc32377ffce47eddf336d53f31945cd71b WatchSource:0}: Error finding container 44ff890feb5c12c309451cf661e1f2fc32377ffce47eddf336d53f31945cd71b: Status 404 returned error can't find the container with id 44ff890feb5c12c309451cf661e1f2fc32377ffce47eddf336d53f31945cd71b
Sep 16 18:43:04.865151 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:43:04.865119    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60eebd82_09af_4aa9_bd33_fd9376cd731d.slice/crio-779bebd3b970a0159be1cce6b56d8150e3159c85fcf27d6602acb466fa00a17c WatchSource:0}: Error finding container 779bebd3b970a0159be1cce6b56d8150e3159c85fcf27d6602acb466fa00a17c: Status 404 returned error can't find the container with id 779bebd3b970a0159be1cce6b56d8150e3159c85fcf27d6602acb466fa00a17c
Sep 16 18:43:04.873798 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.867974    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"]
Sep 16 18:43:04.986271 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.986237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d" event={"ID":"748f92c9-b51e-4266-88c8-719a0dcd5557","Type":"ContainerStarted","Data":"f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412"}
Sep 16 18:43:04.986493 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.986286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d" event={"ID":"748f92c9-b51e-4266-88c8-719a0dcd5557","Type":"ContainerStarted","Data":"709a662e6d28c715c7e4d010a0796bcd5e1dd0077833944298a485c0e5b8f791"}
Sep 16 18:43:04.988056 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.988019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq" event={"ID":"a049b337-43ea-4d64-b797-e00b1178d276","Type":"ContainerStarted","Data":"d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e"}
Sep 16 18:43:04.988209 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.988087    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:04.988209 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.988115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq" event={"ID":"a049b337-43ea-4d64-b797-e00b1178d276","Type":"ContainerStarted","Data":"44ff890feb5c12c309451cf661e1f2fc32377ffce47eddf336d53f31945cd71b"}
Sep 16 18:43:04.989797 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.989773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2" event={"ID":"60eebd82-09af-4aa9-bd33-fd9376cd731d","Type":"ContainerStarted","Data":"967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74"}
Sep 16 18:43:04.989989 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.989811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2" event={"ID":"60eebd82-09af-4aa9-bd33-fd9376cd731d","Type":"ContainerStarted","Data":"779bebd3b970a0159be1cce6b56d8150e3159c85fcf27d6602acb466fa00a17c"}
Sep 16 18:43:04.989989 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:04.989912    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:05.009610 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:05.009564    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d" podStartSLOduration=2.009548139 podStartE2EDuration="2.009548139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:05.008689892 +0000 UTC m=+927.717929286" watchObservedRunningTime="2026-09-16 18:43:05.009548139 +0000 UTC m=+927.718787531"
Sep 16 18:43:05.032239 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:05.032196    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq" podStartSLOduration=1.032180627 podStartE2EDuration="1.032180627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:05.030912817 +0000 UTC m=+927.740152209" watchObservedRunningTime="2026-09-16 18:43:05.032180627 +0000 UTC m=+927.741420018"
Sep 16 18:43:05.052047 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:05.051957    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2" podStartSLOduration=1.051920846 podStartE2EDuration="1.051920846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:05.051188835 +0000 UTC m=+927.760428228" watchObservedRunningTime="2026-09-16 18:43:05.051920846 +0000 UTC m=+927.761160237"
Sep 16 18:43:13.827658 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:13.827622    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:13.828057 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:13.827678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:13.840172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:13.840146    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:14.046548 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:14.046518    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:43:15.997254 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:15.997220    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:15.997780 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:15.997294    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:18.037161 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:18.037123    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"]
Sep 16 18:43:18.039992 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:18.039973    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"
Sep 16 18:43:18.042721 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:18.042697    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:43:18.042840 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:18.042697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:43:18.049343 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:18.049242    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"]
Sep 16 18:43:18.143069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:18.143032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c62be9-1bd5-412f-ae15-7aba4035a30f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0c558678-fbkrd\" (UID: \"24c62be9-1bd5-412f-ae15-7aba4035a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"
Sep 16 18:43:18.244494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:18.244452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c62be9-1bd5-412f-ae15-7aba4035a30f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0c558678-fbkrd\" (UID: \"24c62be9-1bd5-412f-ae15-7aba4035a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"
Sep 16 18:43:18.245236 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:18.245214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c62be9-1bd5-412f-ae15-7aba4035a30f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0c558678-fbkrd\" (UID: \"24c62be9-1bd5-412f-ae15-7aba4035a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"
Sep 16 18:43:18.354311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:18.354237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"
Sep 16 18:43:18.631595 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:18.631522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"]
Sep 16 18:43:18.634272 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:43:18.634241    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c62be9_1bd5_412f_ae15_7aba4035a30f.slice/crio-42d4ce1b49d44555f6c05b02794957080206ffc6991e894f137b5b74f50f72d0 WatchSource:0}: Error finding container 42d4ce1b49d44555f6c05b02794957080206ffc6991e894f137b5b74f50f72d0: Status 404 returned error can't find the container with id 42d4ce1b49d44555f6c05b02794957080206ffc6991e894f137b5b74f50f72d0
Sep 16 18:43:19.069180 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:19.069144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd" event={"ID":"24c62be9-1bd5-412f-ae15-7aba4035a30f","Type":"ContainerStarted","Data":"da62a2258b64052cd672fa2a7644e50d50c2dd9d7d6dae3e1a47ab148173988f"}
Sep 16 18:43:19.069609 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:19.069235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd" event={"ID":"24c62be9-1bd5-412f-ae15-7aba4035a30f","Type":"ContainerStarted","Data":"42d4ce1b49d44555f6c05b02794957080206ffc6991e894f137b5b74f50f72d0"}
Sep 16 18:43:19.087784 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:19.087734    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd" podStartSLOduration=1.087717944 podStartE2EDuration="1.087717944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:19.085805184 +0000 UTC m=+941.795044590" watchObservedRunningTime="2026-09-16 18:43:19.087717944 +0000 UTC m=+941.796957332"
Sep 16 18:43:20.076317 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:20.076286    2580 generic.go:356] "Generic (PLEG): container finished" podID="24c62be9-1bd5-412f-ae15-7aba4035a30f" containerID="da62a2258b64052cd672fa2a7644e50d50c2dd9d7d6dae3e1a47ab148173988f" exitCode=0
Sep 16 18:43:20.076790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:20.076401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd" event={"ID":"24c62be9-1bd5-412f-ae15-7aba4035a30f","Type":"ContainerDied","Data":"da62a2258b64052cd672fa2a7644e50d50c2dd9d7d6dae3e1a47ab148173988f"}
Sep 16 18:43:20.633510 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:20.633474    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:43:21.085887 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:21.085854    2580 generic.go:356] "Generic (PLEG): container finished" podID="40780c93-968c-4147-ac5e-2437983c56d8" containerID="fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959" exitCode=0
Sep 16 18:43:21.086315 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:21.085930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" event={"ID":"40780c93-968c-4147-ac5e-2437983c56d8","Type":"ContainerDied","Data":"fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959"}
Sep 16 18:43:21.086636 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:21.086387    2580 scope.go:117] "RemoveContainer" containerID="fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959"
Sep 16 18:43:21.087614 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:21.087582    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:43:21.087691 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:21.087671    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:43:21.087740 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:21.087725    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:43:21.232009 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:21.231978    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"
Sep 16 18:43:21.375601 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:21.375450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c62be9-1bd5-412f-ae15-7aba4035a30f-maas-api-service-ca\") pod \"24c62be9-1bd5-412f-ae15-7aba4035a30f\" (UID: \"24c62be9-1bd5-412f-ae15-7aba4035a30f\") "
Sep 16 18:43:21.376021 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:21.375970    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c62be9-1bd5-412f-ae15-7aba4035a30f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24c62be9-1bd5-412f-ae15-7aba4035a30f" (UID: "24c62be9-1bd5-412f-ae15-7aba4035a30f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:21.476728 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:21.476686    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c62be9-1bd5-412f-ae15-7aba4035a30f-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:22.092846 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:22.092753    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"
Sep 16 18:43:22.092846 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:22.092803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd" event={"ID":"24c62be9-1bd5-412f-ae15-7aba4035a30f","Type":"ContainerDied","Data":"42d4ce1b49d44555f6c05b02794957080206ffc6991e894f137b5b74f50f72d0"}
Sep 16 18:43:22.093312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:22.092859    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42d4ce1b49d44555f6c05b02794957080206ffc6991e894f137b5b74f50f72d0"
Sep 16 18:43:22.094925 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:22.094880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" event={"ID":"40780c93-968c-4147-ac5e-2437983c56d8","Type":"ContainerStarted","Data":"9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57"}
Sep 16 18:43:22.576848 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:22.576809    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:43:22.577211 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:22.577186    2580 patch_prober.go:28] interesting pod/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 18:43:22.577397 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:22.577368    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 18:43:23.577494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:23.576995    2580 patch_prober.go:28] interesting pod/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 18:43:23.577494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:23.577066    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 18:43:24.577028 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:24.576980    2580 patch_prober.go:28] interesting pod/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 18:43:24.577385 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:24.577314    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 18:43:25.576940 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:25.576898    2580 patch_prober.go:28] interesting pod/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 18:43:25.577470 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:25.576989    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 18:43:26.577112 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:26.577064    2580 patch_prober.go:28] interesting pod/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 18:43:26.577633 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:26.577148    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 18:43:27.581115 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:27.581079    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:43:27.581597 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:27.581460    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:43:27.582373 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:27.582354    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:43:28.228945 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.228906    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"]
Sep 16 18:43:28.229198 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.229170    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq" podUID="a049b337-43ea-4d64-b797-e00b1178d276" containerName="payload-processing" containerID="cri-o://d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e" gracePeriod=30
Sep 16 18:43:28.237601 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.237574    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"]
Sep 16 18:43:28.237932 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.237765    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2" podUID="60eebd82-09af-4aa9-bd33-fd9376cd731d" containerName="payload-pre-processing" containerID="cri-o://967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74" gracePeriod=30
Sep 16 18:43:28.513388 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.513363    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:28.516626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.516607    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:28.615406 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.615376    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"]
Sep 16 18:43:28.618869 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.618847    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0c558678-fbkrd"]
Sep 16 18:43:28.653037 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.652980    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cghr\" (UniqueName: \"kubernetes.io/projected/a049b337-43ea-4d64-b797-e00b1178d276-kube-api-access-6cghr\") pod \"a049b337-43ea-4d64-b797-e00b1178d276\" (UID: \"a049b337-43ea-4d64-b797-e00b1178d276\") "
Sep 16 18:43:28.653037 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.653030    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8lrg\" (UniqueName: \"kubernetes.io/projected/60eebd82-09af-4aa9-bd33-fd9376cd731d-kube-api-access-w8lrg\") pod \"60eebd82-09af-4aa9-bd33-fd9376cd731d\" (UID: \"60eebd82-09af-4aa9-bd33-fd9376cd731d\") "
Sep 16 18:43:28.653203 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.653101    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a049b337-43ea-4d64-b797-e00b1178d276-plugins-config-volume\") pod \"a049b337-43ea-4d64-b797-e00b1178d276\" (UID: \"a049b337-43ea-4d64-b797-e00b1178d276\") "
Sep 16 18:43:28.653203 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.653123    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eebd82-09af-4aa9-bd33-fd9376cd731d-plugins-config-volume\") pod \"60eebd82-09af-4aa9-bd33-fd9376cd731d\" (UID: \"60eebd82-09af-4aa9-bd33-fd9376cd731d\") "
Sep 16 18:43:28.653518 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.653485    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a049b337-43ea-4d64-b797-e00b1178d276-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a049b337-43ea-4d64-b797-e00b1178d276" (UID: "a049b337-43ea-4d64-b797-e00b1178d276"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:28.653628 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.653530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60eebd82-09af-4aa9-bd33-fd9376cd731d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60eebd82-09af-4aa9-bd33-fd9376cd731d" (UID: "60eebd82-09af-4aa9-bd33-fd9376cd731d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:28.655242 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.655223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a049b337-43ea-4d64-b797-e00b1178d276-kube-api-access-6cghr" (OuterVolumeSpecName: "kube-api-access-6cghr") pod "a049b337-43ea-4d64-b797-e00b1178d276" (UID: "a049b337-43ea-4d64-b797-e00b1178d276"). InnerVolumeSpecName "kube-api-access-6cghr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:28.655316 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.655239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60eebd82-09af-4aa9-bd33-fd9376cd731d-kube-api-access-w8lrg" (OuterVolumeSpecName: "kube-api-access-w8lrg") pod "60eebd82-09af-4aa9-bd33-fd9376cd731d" (UID: "60eebd82-09af-4aa9-bd33-fd9376cd731d"). InnerVolumeSpecName "kube-api-access-w8lrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:28.752272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.752171    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"]
Sep 16 18:43:28.754412 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.754385    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cghr\" (UniqueName: \"kubernetes.io/projected/a049b337-43ea-4d64-b797-e00b1178d276-kube-api-access-6cghr\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:28.754412 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.754412    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8lrg\" (UniqueName: \"kubernetes.io/projected/60eebd82-09af-4aa9-bd33-fd9376cd731d-kube-api-access-w8lrg\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:28.754590 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.754422    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a049b337-43ea-4d64-b797-e00b1178d276-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:28.754590 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:28.754431    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eebd82-09af-4aa9-bd33-fd9376cd731d-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:29.130895 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.130856    2580 generic.go:356] "Generic (PLEG): container finished" podID="a049b337-43ea-4d64-b797-e00b1178d276" containerID="d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e" exitCode=0
Sep 16 18:43:29.131079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.130906    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"
Sep 16 18:43:29.131079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.130929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq" event={"ID":"a049b337-43ea-4d64-b797-e00b1178d276","Type":"ContainerDied","Data":"d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e"}
Sep 16 18:43:29.131079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.130983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq" event={"ID":"a049b337-43ea-4d64-b797-e00b1178d276","Type":"ContainerDied","Data":"44ff890feb5c12c309451cf661e1f2fc32377ffce47eddf336d53f31945cd71b"}
Sep 16 18:43:29.131079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.131010    2580 scope.go:117] "RemoveContainer" containerID="d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e"
Sep 16 18:43:29.132466 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.132443    2580 generic.go:356] "Generic (PLEG): container finished" podID="60eebd82-09af-4aa9-bd33-fd9376cd731d" containerID="967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74" exitCode=0
Sep 16 18:43:29.132587 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.132481    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"
Sep 16 18:43:29.132587 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.132533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2" event={"ID":"60eebd82-09af-4aa9-bd33-fd9376cd731d","Type":"ContainerDied","Data":"967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74"}
Sep 16 18:43:29.132587 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.132580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2" event={"ID":"60eebd82-09af-4aa9-bd33-fd9376cd731d","Type":"ContainerDied","Data":"779bebd3b970a0159be1cce6b56d8150e3159c85fcf27d6602acb466fa00a17c"}
Sep 16 18:43:29.141070 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.133416    2580 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" secret="" err="secret \"e2e-derive-0c558678-openshift-default-dockercfg-74f75\" not found"
Sep 16 18:43:29.147431 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.147414    2580 scope.go:117] "RemoveContainer" containerID="d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e"
Sep 16 18:43:29.147702 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:43:29.147683    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e\": container with ID starting with d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e not found: ID does not exist" containerID="d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e"
Sep 16 18:43:29.147751 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.147709    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e"} err="failed to get container status \"d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e\": rpc error: code = NotFound desc = could not find container \"d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e\": container with ID starting with d1c3c4d76ab1681af456314eca284198866d9add44f904a9dd638f1ef01a3c6e not found: ID does not exist"
Sep 16 18:43:29.147751 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.147726    2580 scope.go:117] "RemoveContainer" containerID="967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74"
Sep 16 18:43:29.155421 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.155398    2580 scope.go:117] "RemoveContainer" containerID="967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74"
Sep 16 18:43:29.155699 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:43:29.155681    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74\": container with ID starting with 967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74 not found: ID does not exist" containerID="967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74"
Sep 16 18:43:29.155751 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.155709    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74"} err="failed to get container status \"967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74\": rpc error: code = NotFound desc = could not find container \"967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74\": container with ID starting with 967e468e3311f090394c480718f12d756ba13f8ba22c5aa90bd6666485b4ad74 not found: ID does not exist"
Sep 16 18:43:29.160681 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.160656    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"]
Sep 16 18:43:29.169275 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.169251    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0c558678-6d7cfc8989-8nfbq"]
Sep 16 18:43:29.180885 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.180860    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"]
Sep 16 18:43:29.187026 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.187006    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0c558678-7c67448d9d-nvzf2"]
Sep 16 18:43:29.783144 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.783111    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c62be9-1bd5-412f-ae15-7aba4035a30f" path="/var/lib/kubelet/pods/24c62be9-1bd5-412f-ae15-7aba4035a30f/volumes"
Sep 16 18:43:29.783543 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.783507    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60eebd82-09af-4aa9-bd33-fd9376cd731d" path="/var/lib/kubelet/pods/60eebd82-09af-4aa9-bd33-fd9376cd731d/volumes"
Sep 16 18:43:29.783825 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:29.783813    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a049b337-43ea-4d64-b797-e00b1178d276" path="/var/lib/kubelet/pods/a049b337-43ea-4d64-b797-e00b1178d276/volumes"
Sep 16 18:43:30.139400 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:30.139288    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy" containerID="cri-o://9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57" gracePeriod=30
Sep 16 18:43:35.396586 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.396552    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:43:35.414291 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-credential-socket\") pod \"40780c93-968c-4147-ac5e-2437983c56d8\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") "
Sep 16 18:43:35.414291 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-socket\") pod \"40780c93-968c-4147-ac5e-2437983c56d8\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") "
Sep 16 18:43:35.414481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40780c93-968c-4147-ac5e-2437983c56d8-istiod-ca-cert\") pod \"40780c93-968c-4147-ac5e-2437983c56d8\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") "
Sep 16 18:43:35.414481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414381    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-istio-token\") pod \"40780c93-968c-4147-ac5e-2437983c56d8\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") "
Sep 16 18:43:35.414481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414408    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-data\") pod \"40780c93-968c-4147-ac5e-2437983c56d8\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") "
Sep 16 18:43:35.414640 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9l9x\" (UniqueName: \"kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-kube-api-access-p9l9x\") pod \"40780c93-968c-4147-ac5e-2437983c56d8\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") "
Sep 16 18:43:35.414640 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40780c93-968c-4147-ac5e-2437983c56d8" (UID: "40780c93-968c-4147-ac5e-2437983c56d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:35.414640 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414522    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40780c93-968c-4147-ac5e-2437983c56d8-istio-podinfo\") pod \"40780c93-968c-4147-ac5e-2437983c56d8\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") "
Sep 16 18:43:35.414640 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-certs\") pod \"40780c93-968c-4147-ac5e-2437983c56d8\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") "
Sep 16 18:43:35.414640 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414572    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40780c93-968c-4147-ac5e-2437983c56d8" (UID: "40780c93-968c-4147-ac5e-2437983c56d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:35.414640 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414598    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-envoy\") pod \"40780c93-968c-4147-ac5e-2437983c56d8\" (UID: \"40780c93-968c-4147-ac5e-2437983c56d8\") "
Sep 16 18:43:35.414933 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "40780c93-968c-4147-ac5e-2437983c56d8" (UID: "40780c93-968c-4147-ac5e-2437983c56d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:35.414933 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414858    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40780c93-968c-4147-ac5e-2437983c56d8" (UID: "40780c93-968c-4147-ac5e-2437983c56d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:35.414933 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40780c93-968c-4147-ac5e-2437983c56d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40780c93-968c-4147-ac5e-2437983c56d8" (UID: "40780c93-968c-4147-ac5e-2437983c56d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:35.415044 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414973    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-credential-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:35.415044 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.414991    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:35.415044 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.415005    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40780c93-968c-4147-ac5e-2437983c56d8-istiod-ca-cert\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:35.415044 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.415019    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-data\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:35.415044 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.415030    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-workload-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:35.416702 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.416676    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40780c93-968c-4147-ac5e-2437983c56d8" (UID: "40780c93-968c-4147-ac5e-2437983c56d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:35.416820 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.416672    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "40780c93-968c-4147-ac5e-2437983c56d8" (UID: "40780c93-968c-4147-ac5e-2437983c56d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:35.416820 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.416749    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-kube-api-access-p9l9x" (OuterVolumeSpecName: "kube-api-access-p9l9x") pod "40780c93-968c-4147-ac5e-2437983c56d8" (UID: "40780c93-968c-4147-ac5e-2437983c56d8"). InnerVolumeSpecName "kube-api-access-p9l9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:35.416900 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.416830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40780c93-968c-4147-ac5e-2437983c56d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40780c93-968c-4147-ac5e-2437983c56d8" (UID: "40780c93-968c-4147-ac5e-2437983c56d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:43:35.516255 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.516226    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9l9x\" (UniqueName: \"kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-kube-api-access-p9l9x\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:35.516255 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.516254    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40780c93-968c-4147-ac5e-2437983c56d8-istio-podinfo\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:35.516456 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.516266    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40780c93-968c-4147-ac5e-2437983c56d8-istio-envoy\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:35.516456 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.516274    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40780c93-968c-4147-ac5e-2437983c56d8-istio-token\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:35.738060 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.737983    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"]
Sep 16 18:43:35.738374 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738362    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c62be9-1bd5-412f-ae15-7aba4035a30f" containerName="revoke-keys"
Sep 16 18:43:35.738427 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738376    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c62be9-1bd5-412f-ae15-7aba4035a30f" containerName="revoke-keys"
Sep 16 18:43:35.738427 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738388    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy"
Sep 16 18:43:35.738427 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738394    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy"
Sep 16 18:43:35.738427 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738402    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60eebd82-09af-4aa9-bd33-fd9376cd731d" containerName="payload-pre-processing"
Sep 16 18:43:35.738427 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738407    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60eebd82-09af-4aa9-bd33-fd9376cd731d" containerName="payload-pre-processing"
Sep 16 18:43:35.738427 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738419    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a049b337-43ea-4d64-b797-e00b1178d276" containerName="payload-processing"
Sep 16 18:43:35.738427 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738425    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a049b337-43ea-4d64-b797-e00b1178d276" containerName="payload-processing"
Sep 16 18:43:35.738641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738480    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60eebd82-09af-4aa9-bd33-fd9376cd731d" containerName="payload-pre-processing"
Sep 16 18:43:35.738641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738488    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a049b337-43ea-4d64-b797-e00b1178d276" containerName="payload-processing"
Sep 16 18:43:35.738641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738497    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy"
Sep 16 18:43:35.738641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738503    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c62be9-1bd5-412f-ae15-7aba4035a30f" containerName="revoke-keys"
Sep 16 18:43:35.738641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.738508    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy"
Sep 16 18:43:35.740601 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.740585    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"
Sep 16 18:43:35.743195 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.743174    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:43:35.743310 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.743177    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:43:35.766234 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.766202    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"]
Sep 16 18:43:35.818643 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.818613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/776b5630-7ee3-4cd7-88af-1bf0e1f110ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp\" (UID: \"776b5630-7ee3-4cd7-88af-1bf0e1f110ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"
Sep 16 18:43:35.919443 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.919407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/776b5630-7ee3-4cd7-88af-1bf0e1f110ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp\" (UID: \"776b5630-7ee3-4cd7-88af-1bf0e1f110ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"
Sep 16 18:43:35.920052 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:35.920023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/776b5630-7ee3-4cd7-88af-1bf0e1f110ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp\" (UID: \"776b5630-7ee3-4cd7-88af-1bf0e1f110ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"
Sep 16 18:43:36.050844 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.050816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"
Sep 16 18:43:36.179379 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.179349    2580 generic.go:356] "Generic (PLEG): container finished" podID="40780c93-968c-4147-ac5e-2437983c56d8" containerID="9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57" exitCode=0
Sep 16 18:43:36.179567 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.179426    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"
Sep 16 18:43:36.179567 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.179436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" event={"ID":"40780c93-968c-4147-ac5e-2437983c56d8","Type":"ContainerDied","Data":"9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57"}
Sep 16 18:43:36.179567 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.179494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b" event={"ID":"40780c93-968c-4147-ac5e-2437983c56d8","Type":"ContainerDied","Data":"043c4c67c90320ac006990853990521fa0049cdd5d11d7e7933ca22cd304611f"}
Sep 16 18:43:36.179567 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.179518    2580 scope.go:117] "RemoveContainer" containerID="9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57"
Sep 16 18:43:36.188740 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.188729    2580 scope.go:117] "RemoveContainer" containerID="fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959"
Sep 16 18:43:36.199485 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.199462    2580 scope.go:117] "RemoveContainer" containerID="9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57"
Sep 16 18:43:36.199714 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.199699    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"]
Sep 16 18:43:36.199767 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:43:36.199746    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57\": container with ID starting with 9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57 not found: ID does not exist" containerID="9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57"
Sep 16 18:43:36.199807 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.199771    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57"} err="failed to get container status \"9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57\": rpc error: code = NotFound desc = could not find container \"9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57\": container with ID starting with 9201ce5fa7d71104833aa952e439ab1f8956fd966896919b42674a4c50d78b57 not found: ID does not exist"
Sep 16 18:43:36.199807 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.199789    2580 scope.go:117] "RemoveContainer" containerID="fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959"
Sep 16 18:43:36.200085 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:43:36.200059    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959\": container with ID starting with fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959 not found: ID does not exist" containerID="fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959"
Sep 16 18:43:36.200182 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.200089    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959"} err="failed to get container status \"fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959\": rpc error: code = NotFound desc = could not find container \"fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959\": container with ID starting with fde5b91531a2f8d0a1766206680cf47459d76ac4f06e4afbf55cd0b23bfc8959 not found: ID does not exist"
Sep 16 18:43:36.204656 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.204635    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0c558678-openshift-default-6f8bd4b58d-sbp7b"]
Sep 16 18:43:36.320601 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:36.320533    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"]
Sep 16 18:43:36.323783 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:43:36.323760    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776b5630_7ee3_4cd7_88af_1bf0e1f110ff.slice/crio-2565816fc551ca3671fad95120aba5c501b2f63f8ab4b010b9077f42c25eec13 WatchSource:0}: Error finding container 2565816fc551ca3671fad95120aba5c501b2f63f8ab4b010b9077f42c25eec13: Status 404 returned error can't find the container with id 2565816fc551ca3671fad95120aba5c501b2f63f8ab4b010b9077f42c25eec13
Sep 16 18:43:37.186832 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:37.186799    2580 generic.go:356] "Generic (PLEG): container finished" podID="776b5630-7ee3-4cd7-88af-1bf0e1f110ff" containerID="2dd23e00e35106169010fed7c03c06ea296b5c1c4640e2fa4417ba8cc9369895" exitCode=0
Sep 16 18:43:37.187166 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:37.186835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp" event={"ID":"776b5630-7ee3-4cd7-88af-1bf0e1f110ff","Type":"ContainerDied","Data":"2dd23e00e35106169010fed7c03c06ea296b5c1c4640e2fa4417ba8cc9369895"}
Sep 16 18:43:37.187166 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:37.186873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp" event={"ID":"776b5630-7ee3-4cd7-88af-1bf0e1f110ff","Type":"ContainerStarted","Data":"2565816fc551ca3671fad95120aba5c501b2f63f8ab4b010b9077f42c25eec13"}
Sep 16 18:43:37.436257 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:37.436178    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:43:37.784325 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:37.784292    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40780c93-968c-4147-ac5e-2437983c56d8" path="/var/lib/kubelet/pods/40780c93-968c-4147-ac5e-2437983c56d8/volumes"
Sep 16 18:43:38.329454 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:38.329429    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"
Sep 16 18:43:38.339262 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:38.339228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/776b5630-7ee3-4cd7-88af-1bf0e1f110ff-maas-api-service-ca\") pod \"776b5630-7ee3-4cd7-88af-1bf0e1f110ff\" (UID: \"776b5630-7ee3-4cd7-88af-1bf0e1f110ff\") "
Sep 16 18:43:38.339575 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:38.339552    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776b5630-7ee3-4cd7-88af-1bf0e1f110ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "776b5630-7ee3-4cd7-88af-1bf0e1f110ff" (UID: "776b5630-7ee3-4cd7-88af-1bf0e1f110ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:38.440668 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:38.440635    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/776b5630-7ee3-4cd7-88af-1bf0e1f110ff-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:39.197890 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:39.197862    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"
Sep 16 18:43:39.198057 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:39.197909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp" event={"ID":"776b5630-7ee3-4cd7-88af-1bf0e1f110ff","Type":"ContainerDied","Data":"2565816fc551ca3671fad95120aba5c501b2f63f8ab4b010b9077f42c25eec13"}
Sep 16 18:43:39.198057 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:39.197956    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2565816fc551ca3671fad95120aba5c501b2f63f8ab4b010b9077f42c25eec13"
Sep 16 18:43:45.882565 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:45.882522    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"]
Sep 16 18:43:45.882914 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:45.882764    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6" podUID="f8ab76a8-a0e0-4316-bf84-b94ea1824836" containerName="payload-processing" containerID="cri-o://26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97" gracePeriod=30
Sep 16 18:43:46.134520 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.134468    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:43:46.214315 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.214288    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ab76a8-a0e0-4316-bf84-b94ea1824836-plugins-config-volume\") pod \"f8ab76a8-a0e0-4316-bf84-b94ea1824836\" (UID: \"f8ab76a8-a0e0-4316-bf84-b94ea1824836\") "
Sep 16 18:43:46.214473 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.214400    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdgt5\" (UniqueName: \"kubernetes.io/projected/f8ab76a8-a0e0-4316-bf84-b94ea1824836-kube-api-access-vdgt5\") pod \"f8ab76a8-a0e0-4316-bf84-b94ea1824836\" (UID: \"f8ab76a8-a0e0-4316-bf84-b94ea1824836\") "
Sep 16 18:43:46.214626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.214603    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ab76a8-a0e0-4316-bf84-b94ea1824836-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8ab76a8-a0e0-4316-bf84-b94ea1824836" (UID: "f8ab76a8-a0e0-4316-bf84-b94ea1824836"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:46.216447 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.216427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ab76a8-a0e0-4316-bf84-b94ea1824836-kube-api-access-vdgt5" (OuterVolumeSpecName: "kube-api-access-vdgt5") pod "f8ab76a8-a0e0-4316-bf84-b94ea1824836" (UID: "f8ab76a8-a0e0-4316-bf84-b94ea1824836"). InnerVolumeSpecName "kube-api-access-vdgt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:46.233010 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.232978    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8ab76a8-a0e0-4316-bf84-b94ea1824836" containerID="26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97" exitCode=0
Sep 16 18:43:46.233118 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.233024    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"
Sep 16 18:43:46.233118 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.233058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6" event={"ID":"f8ab76a8-a0e0-4316-bf84-b94ea1824836","Type":"ContainerDied","Data":"26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97"}
Sep 16 18:43:46.233118 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.233108    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6" event={"ID":"f8ab76a8-a0e0-4316-bf84-b94ea1824836","Type":"ContainerDied","Data":"821cd27a751c1a5cdad07601b09753c8e7add7aafc8ef0139ccffbc28a3937f8"}
Sep 16 18:43:46.233235 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.233131    2580 scope.go:117] "RemoveContainer" containerID="26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97"
Sep 16 18:43:46.242168 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.242146    2580 scope.go:117] "RemoveContainer" containerID="26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97"
Sep 16 18:43:46.242594 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:43:46.242566    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97\": container with ID starting with 26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97 not found: ID does not exist" containerID="26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97"
Sep 16 18:43:46.242682 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.242602    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97"} err="failed to get container status \"26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97\": rpc error: code = NotFound desc = could not find container \"26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97\": container with ID starting with 26f34a21c6bb3cc1f6090f65abdec8095fbc6da2657c79b210edbe415bef4b97 not found: ID does not exist"
Sep 16 18:43:46.244199 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.244179    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"]
Sep 16 18:43:46.247481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.247461    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-90ea7d-qx6gp"]
Sep 16 18:43:46.257042 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.257019    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"]
Sep 16 18:43:46.261399 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.261377    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-90ea7d-6cff96c66f-jh8c6"]
Sep 16 18:43:46.315810 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.315789    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ab76a8-a0e0-4316-bf84-b94ea1824836-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:46.315810 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:46.315809    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdgt5\" (UniqueName: \"kubernetes.io/projected/f8ab76a8-a0e0-4316-bf84-b94ea1824836-kube-api-access-vdgt5\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:43:47.783605 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:47.783568    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776b5630-7ee3-4cd7-88af-1bf0e1f110ff" path="/var/lib/kubelet/pods/776b5630-7ee3-4cd7-88af-1bf0e1f110ff/volumes"
Sep 16 18:43:47.784145 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:47.784123    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8ab76a8-a0e0-4316-bf84-b94ea1824836" path="/var/lib/kubelet/pods/f8ab76a8-a0e0-4316-bf84-b94ea1824836/volumes"
Sep 16 18:43:51.670249 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:51.670216    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"]
Sep 16 18:43:52.824202 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:43:52.824164    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:44:06.689587 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:06.689498    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d" podUID="748f92c9-b51e-4266-88c8-719a0dcd5557" containerName="main" containerID="cri-o://f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412" gracePeriod=45
Sep 16 18:44:06.942975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:06.942917    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:44:07.114267 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.114235    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/748f92c9-b51e-4266-88c8-719a0dcd5557-tls-certs\") pod \"748f92c9-b51e-4266-88c8-719a0dcd5557\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") "
Sep 16 18:44:07.114267 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.114272    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-home\") pod \"748f92c9-b51e-4266-88c8-719a0dcd5557\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") "
Sep 16 18:44:07.114517 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.114290    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-tmp-dir\") pod \"748f92c9-b51e-4266-88c8-719a0dcd5557\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") "
Sep 16 18:44:07.114517 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.114311    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-dshm\") pod \"748f92c9-b51e-4266-88c8-719a0dcd5557\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") "
Sep 16 18:44:07.114517 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.114346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-model-cache\") pod \"748f92c9-b51e-4266-88c8-719a0dcd5557\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") "
Sep 16 18:44:07.114517 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.114389    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9vzg\" (UniqueName: \"kubernetes.io/projected/748f92c9-b51e-4266-88c8-719a0dcd5557-kube-api-access-f9vzg\") pod \"748f92c9-b51e-4266-88c8-719a0dcd5557\" (UID: \"748f92c9-b51e-4266-88c8-719a0dcd5557\") "
Sep 16 18:44:07.114735 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.114576    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "748f92c9-b51e-4266-88c8-719a0dcd5557" (UID: "748f92c9-b51e-4266-88c8-719a0dcd5557"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:07.114735 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.114591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-home" (OuterVolumeSpecName: "home") pod "748f92c9-b51e-4266-88c8-719a0dcd5557" (UID: "748f92c9-b51e-4266-88c8-719a0dcd5557"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:07.114735 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.114663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-model-cache" (OuterVolumeSpecName: "model-cache") pod "748f92c9-b51e-4266-88c8-719a0dcd5557" (UID: "748f92c9-b51e-4266-88c8-719a0dcd5557"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:07.116439 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.116415    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-dshm" (OuterVolumeSpecName: "dshm") pod "748f92c9-b51e-4266-88c8-719a0dcd5557" (UID: "748f92c9-b51e-4266-88c8-719a0dcd5557"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:07.116439 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.116424    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/748f92c9-b51e-4266-88c8-719a0dcd5557-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "748f92c9-b51e-4266-88c8-719a0dcd5557" (UID: "748f92c9-b51e-4266-88c8-719a0dcd5557"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:07.116589 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.116470    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748f92c9-b51e-4266-88c8-719a0dcd5557-kube-api-access-f9vzg" (OuterVolumeSpecName: "kube-api-access-f9vzg") pod "748f92c9-b51e-4266-88c8-719a0dcd5557" (UID: "748f92c9-b51e-4266-88c8-719a0dcd5557"). InnerVolumeSpecName "kube-api-access-f9vzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:07.215712 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.215641    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/748f92c9-b51e-4266-88c8-719a0dcd5557-tls-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:44:07.215712 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.215668    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-home\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:44:07.215712 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.215696    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-tmp-dir\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:44:07.215712 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.215703    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-dshm\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:44:07.215712 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.215712    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/748f92c9-b51e-4266-88c8-719a0dcd5557-model-cache\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:44:07.215967 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.215720    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9vzg\" (UniqueName: \"kubernetes.io/projected/748f92c9-b51e-4266-88c8-719a0dcd5557-kube-api-access-f9vzg\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:44:07.344100 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.344070    2580 generic.go:356] "Generic (PLEG): container finished" podID="748f92c9-b51e-4266-88c8-719a0dcd5557" containerID="f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412" exitCode=0
Sep 16 18:44:07.344231 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.344103    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"
Sep 16 18:44:07.344231 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.344140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d" event={"ID":"748f92c9-b51e-4266-88c8-719a0dcd5557","Type":"ContainerDied","Data":"f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412"}
Sep 16 18:44:07.344231 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.344185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d" event={"ID":"748f92c9-b51e-4266-88c8-719a0dcd5557","Type":"ContainerDied","Data":"709a662e6d28c715c7e4d010a0796bcd5e1dd0077833944298a485c0e5b8f791"}
Sep 16 18:44:07.344231 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.344205    2580 scope.go:117] "RemoveContainer" containerID="f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412"
Sep 16 18:44:07.353498 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.353486    2580 scope.go:117] "RemoveContainer" containerID="f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412"
Sep 16 18:44:07.353762 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:44:07.353744    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412\": container with ID starting with f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412 not found: ID does not exist" containerID="f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412"
Sep 16 18:44:07.353811 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.353769    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412"} err="failed to get container status \"f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412\": rpc error: code = NotFound desc = could not find container \"f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412\": container with ID starting with f023247580f41b4b917308a258d0e8e4ac3088d9fecc27441e5002fa4bb83412 not found: ID does not exist"
Sep 16 18:44:07.366677 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.366653    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"]
Sep 16 18:44:07.368874 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.368851    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-90ea7d/autores-model-90ea7d-kserve-7674785d7f-gff6d"]
Sep 16 18:44:07.782623 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:07.782590    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="748f92c9-b51e-4266-88c8-719a0dcd5557" path="/var/lib/kubelet/pods/748f92c9-b51e-4266-88c8-719a0dcd5557/volumes"
Sep 16 18:44:25.236319 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236285    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"]
Sep 16 18:44:25.236737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236660    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ab76a8-a0e0-4316-bf84-b94ea1824836" containerName="payload-processing"
Sep 16 18:44:25.236737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236673    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ab76a8-a0e0-4316-bf84-b94ea1824836" containerName="payload-processing"
Sep 16 18:44:25.236737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236688    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy"
Sep 16 18:44:25.236737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236693    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40780c93-968c-4147-ac5e-2437983c56d8" containerName="istio-proxy"
Sep 16 18:44:25.236737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236702    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="748f92c9-b51e-4266-88c8-719a0dcd5557" containerName="main"
Sep 16 18:44:25.236737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236709    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="748f92c9-b51e-4266-88c8-719a0dcd5557" containerName="main"
Sep 16 18:44:25.236737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236722    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776b5630-7ee3-4cd7-88af-1bf0e1f110ff" containerName="revoke-keys"
Sep 16 18:44:25.236737 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236727    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="776b5630-7ee3-4cd7-88af-1bf0e1f110ff" containerName="revoke-keys"
Sep 16 18:44:25.236986 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236777    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="776b5630-7ee3-4cd7-88af-1bf0e1f110ff" containerName="revoke-keys"
Sep 16 18:44:25.236986 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236785    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="748f92c9-b51e-4266-88c8-719a0dcd5557" containerName="main"
Sep 16 18:44:25.236986 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.236795    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ab76a8-a0e0-4316-bf84-b94ea1824836" containerName="payload-processing"
Sep 16 18:44:25.240217 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.240199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.242889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.242862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-662432\"/\"premium-w4-662432-kserve-self-signed-certs\""
Sep 16 18:44:25.250138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.250117    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"]
Sep 16 18:44:25.271779 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.271752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-dshm\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.271912 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.271803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-home\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.271912 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.271824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-model-cache\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.272010 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.271924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bws4n\" (UniqueName: \"kubernetes.io/projected/1737c29b-2643-4d61-8678-7222756a5a93-kube-api-access-bws4n\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.272010 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.271972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1737c29b-2643-4d61-8678-7222756a5a93-tls-certs\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.272129 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.272071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-tmp-dir\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.372999 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.372898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bws4n\" (UniqueName: \"kubernetes.io/projected/1737c29b-2643-4d61-8678-7222756a5a93-kube-api-access-bws4n\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.372999 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.372950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1737c29b-2643-4d61-8678-7222756a5a93-tls-certs\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.372999 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.372999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-tmp-dir\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.373299 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.373042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-dshm\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.373299 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.373074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-home\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.373299 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.373096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-model-cache\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.373497 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.373472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-tmp-dir\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.373566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.373509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-model-cache\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.373618 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.373560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-home\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.375436 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.375408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-dshm\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.375541 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.375451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1737c29b-2643-4d61-8678-7222756a5a93-tls-certs\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.381413 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.381394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bws4n\" (UniqueName: \"kubernetes.io/projected/1737c29b-2643-4d61-8678-7222756a5a93-kube-api-access-bws4n\") pod \"premium-w4-662432-kserve-5b5bffbfc-gkfdx\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") " pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.549827 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.549794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:25.827480 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:25.827452    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"]
Sep 16 18:44:25.829526 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:44:25.829497    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1737c29b_2643_4d61_8678_7222756a5a93.slice/crio-4ec9225bf0f2332cea21f6f646be668d8f71e15d48cab384a59057366b006521 WatchSource:0}: Error finding container 4ec9225bf0f2332cea21f6f646be668d8f71e15d48cab384a59057366b006521: Status 404 returned error can't find the container with id 4ec9225bf0f2332cea21f6f646be668d8f71e15d48cab384a59057366b006521
Sep 16 18:44:26.444722 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:26.444686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx" event={"ID":"1737c29b-2643-4d61-8678-7222756a5a93","Type":"ContainerStarted","Data":"32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191"}
Sep 16 18:44:26.444722 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:26.444726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx" event={"ID":"1737c29b-2643-4d61-8678-7222756a5a93","Type":"ContainerStarted","Data":"4ec9225bf0f2332cea21f6f646be668d8f71e15d48cab384a59057366b006521"}
Sep 16 18:44:26.465095 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:26.465042    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx" podStartSLOduration=1.4650275609999999 podStartE2EDuration="1.465027561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:26.464149333 +0000 UTC m=+1009.173388719" watchObservedRunningTime="2026-09-16 18:44:26.465027561 +0000 UTC m=+1009.174266965"
Sep 16 18:44:30.433709 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:30.433676    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:44:35.549995 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:35.549961    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:35.550453 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:35.550020    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:35.563741 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:35.563711    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:36.506156 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:36.506128    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:44:38.144523 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.144440    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"]
Sep 16 18:44:38.149097 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.149078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.152978 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.152941    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4dda98\"/\"kube-root-ca.crt\""
Sep 16 18:44:38.152978 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.152965    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4dda98\"/\"openshift-service-ca.crt\""
Sep 16 18:44:38.152978 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.152969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4dda98\"/\"default-dockercfg-xgntv\""
Sep 16 18:44:38.153203 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.153035    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4dda98\"/\"premium-w0-4dda98-kserve-self-signed-certs\""
Sep 16 18:44:38.167409 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.167381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"]
Sep 16 18:44:38.288209 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.288179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-home\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.288391 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.288222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tmp-dir\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.288391 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.288324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tls-certs\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.288391 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.288383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-model-cache\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.288580 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.288410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd2vh\" (UniqueName: \"kubernetes.io/projected/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-kube-api-access-fd2vh\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.288580 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.288432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-dshm\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.389718 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.389683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tmp-dir\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.389860 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.389732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tls-certs\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.389860 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.389750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-model-cache\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.389860 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.389771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd2vh\" (UniqueName: \"kubernetes.io/projected/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-kube-api-access-fd2vh\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.389860 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.389790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-dshm\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.389860 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.389834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-home\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.390173 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.390148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tmp-dir\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.390210 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.390183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-model-cache\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.390245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.390228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-home\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.392038 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.392019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-dshm\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.392303 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.392288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tls-certs\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.400272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.400218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd2vh\" (UniqueName: \"kubernetes.io/projected/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-kube-api-access-fd2vh\") pod \"premium-w0-4dda98-kserve-8f54858fd-g9qcs\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") " pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.458162 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.458138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:38.755188 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:38.755103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"]
Sep 16 18:44:38.758359 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:44:38.758302    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7d7ce67_53f9_492f_9713_d319cd5ddeb6.slice/crio-ce6f5df940f08548c16fe2def2b3debbbc407c4739f584c00fd66d3f1da0fa1d WatchSource:0}: Error finding container ce6f5df940f08548c16fe2def2b3debbbc407c4739f584c00fd66d3f1da0fa1d: Status 404 returned error can't find the container with id ce6f5df940f08548c16fe2def2b3debbbc407c4739f584c00fd66d3f1da0fa1d
Sep 16 18:44:39.511042 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:39.511011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs" event={"ID":"f7d7ce67-53f9-492f-9713-d319cd5ddeb6","Type":"ContainerStarted","Data":"7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c"}
Sep 16 18:44:39.511462 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:39.511053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs" event={"ID":"f7d7ce67-53f9-492f-9713-d319cd5ddeb6","Type":"ContainerStarted","Data":"ce6f5df940f08548c16fe2def2b3debbbc407c4739f584c00fd66d3f1da0fa1d"}
Sep 16 18:44:39.532625 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:39.532578    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs" podStartSLOduration=1.53256221 podStartE2EDuration="1.53256221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:39.531435764 +0000 UTC m=+1022.240675156" watchObservedRunningTime="2026-09-16 18:44:39.53256221 +0000 UTC m=+1022.241801601"
Sep 16 18:44:40.187470 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.187430    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"]
Sep 16 18:44:40.191241 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.191221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.193830 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.193807    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-16ae98-openshift-default-dockercfg-kdwrp\""
Sep 16 18:44:40.193968 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.193838    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:44:40.210764 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.210742    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"]
Sep 16 18:44:40.310810 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.310742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-socket\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.310961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.310802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrbsj\" (UniqueName: \"kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-kube-api-access-mrbsj\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.311036 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.310984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-token\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.311036 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.311022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b77c1a-cc53-4a16-b737-abbe7eec3103-istiod-ca-cert\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.311132 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.311094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-certs\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.311187 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.311138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-podinfo\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.311228 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.311169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-envoy\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.311518 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.311273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-credential-socket\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.311553 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.311541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-data\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.412876 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.412849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-socket\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413035 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.412881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrbsj\" (UniqueName: \"kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-kube-api-access-mrbsj\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413035 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.412936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-token\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413035 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.412950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b77c1a-cc53-4a16-b737-abbe7eec3103-istiod-ca-cert\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413035 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.412971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-certs\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413035 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.413024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-podinfo\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413307 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.413064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-envoy\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413307 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.413113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-credential-socket\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413307 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.413242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-socket\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413499 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.413302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-certs\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413499 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.413347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-data\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.413521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-credential-socket\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413719 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.413696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-data\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.413842 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.413813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b77c1a-cc53-4a16-b737-abbe7eec3103-istiod-ca-cert\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.415388 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.415361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-envoy\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.415588 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.415572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-podinfo\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.422284 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.422251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrbsj\" (UniqueName: \"kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-kube-api-access-mrbsj\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.423378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.423358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-token\") pod \"e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") " pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.501771 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.501684    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:40.780963 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.780936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"]
Sep 16 18:44:40.783250 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:44:40.783216    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b77c1a_cc53_4a16_b737_abbe7eec3103.slice/crio-37023d3424893d640c2947e547b87dbf628b490f232c282e0a3380cffcaf786c WatchSource:0}: Error finding container 37023d3424893d640c2947e547b87dbf628b490f232c282e0a3380cffcaf786c: Status 404 returned error can't find the container with id 37023d3424893d640c2947e547b87dbf628b490f232c282e0a3380cffcaf786c
Sep 16 18:44:40.786005 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.785977    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:44:40.786068 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.786041    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:44:40.786108 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:40.786068    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:44:41.523408 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:41.523373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" event={"ID":"93b77c1a-cc53-4a16-b737-abbe7eec3103","Type":"ContainerStarted","Data":"412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb"}
Sep 16 18:44:41.523604 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:41.523439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" event={"ID":"93b77c1a-cc53-4a16-b737-abbe7eec3103","Type":"ContainerStarted","Data":"37023d3424893d640c2947e547b87dbf628b490f232c282e0a3380cffcaf786c"}
Sep 16 18:44:41.548106 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:41.548047    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" podStartSLOduration=1.5480248620000001 podStartE2EDuration="1.548024862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:41.544965948 +0000 UTC m=+1024.254205339" watchObservedRunningTime="2026-09-16 18:44:41.548024862 +0000 UTC m=+1024.257264255"
Sep 16 18:44:42.502649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:42.502614    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:42.508119 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:42.508096    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:42.528378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:42.528348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:42.529103 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:42.529085    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:44:46.939290 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:46.939258    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:44:47.488672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.488634    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"]
Sep 16 18:44:47.492260 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.492239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:44:47.494742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.494719    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-16ae98\""
Sep 16 18:44:47.494742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.494739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-16ae98-dockercfg-qpskl\""
Sep 16 18:44:47.505747 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.505726    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"]
Sep 16 18:44:47.571521 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.571486    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"]
Sep 16 18:44:47.575438 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.575416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:44:47.583703 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.583515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v2j2\" (UniqueName: \"kubernetes.io/projected/95ebdf24-000f-4272-b5bb-d02cb2086c4f-kube-api-access-5v2j2\") pod \"payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246\" (UID: \"95ebdf24-000f-4272-b5bb-d02cb2086c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:44:47.583703 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.583556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ebdf24-000f-4272-b5bb-d02cb2086c4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246\" (UID: \"95ebdf24-000f-4272-b5bb-d02cb2086c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:44:47.589986 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.589946    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"]
Sep 16 18:44:47.684819 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.684778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlpc\" (UniqueName: \"kubernetes.io/projected/2dbf790b-2c71-4728-a6ad-0aededbf37b8-kube-api-access-7vlpc\") pod \"payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj\" (UID: \"2dbf790b-2c71-4728-a6ad-0aededbf37b8\") " pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:44:47.684992 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.684825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v2j2\" (UniqueName: \"kubernetes.io/projected/95ebdf24-000f-4272-b5bb-d02cb2086c4f-kube-api-access-5v2j2\") pod \"payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246\" (UID: \"95ebdf24-000f-4272-b5bb-d02cb2086c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:44:47.684992 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.684848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ebdf24-000f-4272-b5bb-d02cb2086c4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246\" (UID: \"95ebdf24-000f-4272-b5bb-d02cb2086c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:44:47.684992 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.684951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbf790b-2c71-4728-a6ad-0aededbf37b8-plugins-config-volume\") pod \"payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj\" (UID: \"2dbf790b-2c71-4728-a6ad-0aededbf37b8\") " pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:44:47.685937 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.685906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ebdf24-000f-4272-b5bb-d02cb2086c4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246\" (UID: \"95ebdf24-000f-4272-b5bb-d02cb2086c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:44:47.698211 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.698184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v2j2\" (UniqueName: \"kubernetes.io/projected/95ebdf24-000f-4272-b5bb-d02cb2086c4f-kube-api-access-5v2j2\") pod \"payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246\" (UID: \"95ebdf24-000f-4272-b5bb-d02cb2086c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:44:47.786486 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.786444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlpc\" (UniqueName: \"kubernetes.io/projected/2dbf790b-2c71-4728-a6ad-0aededbf37b8-kube-api-access-7vlpc\") pod \"payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj\" (UID: \"2dbf790b-2c71-4728-a6ad-0aededbf37b8\") " pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:44:47.786691 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.786569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbf790b-2c71-4728-a6ad-0aededbf37b8-plugins-config-volume\") pod \"payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj\" (UID: \"2dbf790b-2c71-4728-a6ad-0aededbf37b8\") " pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:44:47.787248 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.787222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbf790b-2c71-4728-a6ad-0aededbf37b8-plugins-config-volume\") pod \"payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj\" (UID: \"2dbf790b-2c71-4728-a6ad-0aededbf37b8\") " pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:44:47.795473 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.795450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlpc\" (UniqueName: \"kubernetes.io/projected/2dbf790b-2c71-4728-a6ad-0aededbf37b8-kube-api-access-7vlpc\") pod \"payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj\" (UID: \"2dbf790b-2c71-4728-a6ad-0aededbf37b8\") " pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:44:47.805245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.805221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:44:47.888577 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:47.888548    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:44:48.078862 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.078837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"]
Sep 16 18:44:48.081139 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:44:48.081110    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ebdf24_000f_4272_b5bb_d02cb2086c4f.slice/crio-23bc6864fc952c913804616ca990d1478d0930d0bd583837482a881e2ab94171 WatchSource:0}: Error finding container 23bc6864fc952c913804616ca990d1478d0930d0bd583837482a881e2ab94171: Status 404 returned error can't find the container with id 23bc6864fc952c913804616ca990d1478d0930d0bd583837482a881e2ab94171
Sep 16 18:44:48.163421 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.163397    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"]
Sep 16 18:44:48.165908 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:44:48.165883    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbf790b_2c71_4728_a6ad_0aededbf37b8.slice/crio-6088cb215950ee4ce50c49f8f788d1f5ff06468bb82859c310daf1333def8015 WatchSource:0}: Error finding container 6088cb215950ee4ce50c49f8f788d1f5ff06468bb82859c310daf1333def8015: Status 404 returned error can't find the container with id 6088cb215950ee4ce50c49f8f788d1f5ff06468bb82859c310daf1333def8015
Sep 16 18:44:48.458660 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.458533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:48.458660 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.458613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:48.473489 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.473461    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:48.563063 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.563024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246" event={"ID":"95ebdf24-000f-4272-b5bb-d02cb2086c4f","Type":"ContainerStarted","Data":"0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661"}
Sep 16 18:44:48.563429 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.563092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246" event={"ID":"95ebdf24-000f-4272-b5bb-d02cb2086c4f","Type":"ContainerStarted","Data":"23bc6864fc952c913804616ca990d1478d0930d0bd583837482a881e2ab94171"}
Sep 16 18:44:48.563429 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.563120    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:44:48.565194 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.564737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj" event={"ID":"2dbf790b-2c71-4728-a6ad-0aededbf37b8","Type":"ContainerStarted","Data":"72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60"}
Sep 16 18:44:48.565194 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.564781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj" event={"ID":"2dbf790b-2c71-4728-a6ad-0aededbf37b8","Type":"ContainerStarted","Data":"6088cb215950ee4ce50c49f8f788d1f5ff06468bb82859c310daf1333def8015"}
Sep 16 18:44:48.565194 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.564929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:44:48.576842 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.576819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:44:48.582578 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.582520    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246" podStartSLOduration=1.582499856 podStartE2EDuration="1.582499856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:48.580347853 +0000 UTC m=+1031.289587240" watchObservedRunningTime="2026-09-16 18:44:48.582499856 +0000 UTC m=+1031.291739249"
Sep 16 18:44:48.623916 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:48.623872    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj" podStartSLOduration=1.623856116 podStartE2EDuration="1.623856116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:48.621614808 +0000 UTC m=+1031.330854201" watchObservedRunningTime="2026-09-16 18:44:48.623856116 +0000 UTC m=+1031.333095537"
Sep 16 18:44:59.573127 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:59.573091    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:44:59.573616 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:44:59.573348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:45:00.167128 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.167088    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x"]
Sep 16 18:45:00.171391 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.171368    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp"]
Sep 16 18:45:00.171537 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.171495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x"
Sep 16 18:45:00.174628 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.174605    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp"
Sep 16 18:45:00.175276 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.175260    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:45:00.181777 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.181759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x"]
Sep 16 18:45:00.184630 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.184613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x"
Sep 16 18:45:00.193616 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.193594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp"
Sep 16 18:45:00.193743 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.193609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp"]
Sep 16 18:45:00.199182 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.198465    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5"]
Sep 16 18:45:00.203032 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.202781    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5"
Sep 16 18:45:00.203746 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.203223    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7"]
Sep 16 18:45:00.207158 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.207138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7"
Sep 16 18:45:00.219290 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.219255    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5"]
Sep 16 18:45:00.221717 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.221674    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7"]
Sep 16 18:45:00.248921 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.248364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5"
Sep 16 18:45:00.258928 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.258602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7"
Sep 16 18:45:00.469066 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.468995    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"]
Sep 16 18:45:00.473688 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.473669    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.479272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.479251    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e8510f\"/\"kube-root-ca.crt\""
Sep 16 18:45:00.480114 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.480096    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e8510f\"/\"default-dockercfg-x2rvt\""
Sep 16 18:45:00.480726 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.480282    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e8510f\"/\"test-model-e8510f-kserve-self-signed-certs\""
Sep 16 18:45:00.480726 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.480525    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e8510f\"/\"openshift-service-ca.crt\""
Sep 16 18:45:00.489228 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.489208    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"]
Sep 16 18:45:00.499487 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.499408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x"]
Sep 16 18:45:00.502216 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:45:00.502190    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b4b104_0a71_41bd_91b1_8cdceda29a8b.slice/crio-2118ca50f744f26804fbeed4b2b38d28ec9151d020a3d7bfe21f208fca8d8254 WatchSource:0}: Error finding container 2118ca50f744f26804fbeed4b2b38d28ec9151d020a3d7bfe21f208fca8d8254: Status 404 returned error can't find the container with id 2118ca50f744f26804fbeed4b2b38d28ec9151d020a3d7bfe21f208fca8d8254
Sep 16 18:45:00.515172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.514896    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-home\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.515172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.514956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv27w\" (UniqueName: \"kubernetes.io/projected/2fe9a490-1875-483c-bc61-0c2af3c83b34-kube-api-access-fv27w\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.515172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.515043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-model-cache\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.515172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.515096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe9a490-1875-483c-bc61-0c2af3c83b34-tls-certs\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.515172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.515131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-dshm\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.515516 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.515215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-tmp-dir\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.515854 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.515766    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp"]
Sep 16 18:45:00.575831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.575807    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7"]
Sep 16 18:45:00.577382 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:45:00.577359    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130a948c_0e38_4cae_bf51_c10b332bf1a5.slice/crio-fb87b834450f94e3fce683369380a8bf020c371712e8de188bd1bb165b8e4e18 WatchSource:0}: Error finding container fb87b834450f94e3fce683369380a8bf020c371712e8de188bd1bb165b8e4e18: Status 404 returned error can't find the container with id fb87b834450f94e3fce683369380a8bf020c371712e8de188bd1bb165b8e4e18
Sep 16 18:45:00.602844 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.602825    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5"]
Sep 16 18:45:00.609319 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:45:00.609297    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224064a5_060e_4ff9_bf02_cc1b10971d35.slice/crio-b8d38dd31c5c893066bb46b388fd2810b8541204bd1c10d874b829a97d028bd7 WatchSource:0}: Error finding container b8d38dd31c5c893066bb46b388fd2810b8541204bd1c10d874b829a97d028bd7: Status 404 returned error can't find the container with id b8d38dd31c5c893066bb46b388fd2810b8541204bd1c10d874b829a97d028bd7
Sep 16 18:45:00.616168 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.616146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-home\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.616231 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.616191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv27w\" (UniqueName: \"kubernetes.io/projected/2fe9a490-1875-483c-bc61-0c2af3c83b34-kube-api-access-fv27w\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.616368 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.616233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-model-cache\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.616368 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.616282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe9a490-1875-483c-bc61-0c2af3c83b34-tls-certs\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.616368 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.616309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-dshm\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.616368 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.616355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-tmp-dir\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.616662 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.616596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-home\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.616725 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.616623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-model-cache\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.616778 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.616752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-tmp-dir\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.618514 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.618492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-dshm\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.618728 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.618712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe9a490-1875-483c-bc61-0c2af3c83b34-tls-certs\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.627701 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.627677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv27w\" (UniqueName: \"kubernetes.io/projected/2fe9a490-1875-483c-bc61-0c2af3c83b34-kube-api-access-fv27w\") pod \"test-model-e8510f-kserve-56757f7bcf-dfvwq\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") " pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:00.630072 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.630012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5" event={"ID":"224064a5-060e-4ff9-bf02-cc1b10971d35","Type":"ContainerStarted","Data":"b8d38dd31c5c893066bb46b388fd2810b8541204bd1c10d874b829a97d028bd7"}
Sep 16 18:45:00.631377 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.631356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x" event={"ID":"a0b4b104-0a71-41bd-91b1-8cdceda29a8b","Type":"ContainerStarted","Data":"2118ca50f744f26804fbeed4b2b38d28ec9151d020a3d7bfe21f208fca8d8254"}
Sep 16 18:45:00.632618 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.632597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7" event={"ID":"130a948c-0e38-4cae-bf51-c10b332bf1a5","Type":"ContainerStarted","Data":"fb87b834450f94e3fce683369380a8bf020c371712e8de188bd1bb165b8e4e18"}
Sep 16 18:45:00.633872 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.633852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp" event={"ID":"5c0df27b-d20a-4d5b-843a-327145560f55","Type":"ContainerStarted","Data":"e78951a2d04b6f994f3809bd36cf5fc20b2eb2d576cee50b914cbf471bd50599"}
Sep 16 18:45:00.782895 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:00.782865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:01.059453 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.059424    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"]
Sep 16 18:45:01.061960 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:45:01.061920    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fe9a490_1875_483c_bc61_0c2af3c83b34.slice/crio-157f83a051fde53f8f47813e57ac5f16b3bc6026d5415780c3420a2c7bd6758b WatchSource:0}: Error finding container 157f83a051fde53f8f47813e57ac5f16b3bc6026d5415780c3420a2c7bd6758b: Status 404 returned error can't find the container with id 157f83a051fde53f8f47813e57ac5f16b3bc6026d5415780c3420a2c7bd6758b
Sep 16 18:45:01.644357 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.642766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq" event={"ID":"2fe9a490-1875-483c-bc61-0c2af3c83b34","Type":"ContainerStarted","Data":"845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b"}
Sep 16 18:45:01.644357 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.642834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq" event={"ID":"2fe9a490-1875-483c-bc61-0c2af3c83b34","Type":"ContainerStarted","Data":"157f83a051fde53f8f47813e57ac5f16b3bc6026d5415780c3420a2c7bd6758b"}
Sep 16 18:45:01.646526 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.646499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp" event={"ID":"5c0df27b-d20a-4d5b-843a-327145560f55","Type":"ContainerStarted","Data":"cde9d301ef5ebea3005175b7ad9a8629144d3fcd1f85b1915e21550e1836323e"}
Sep 16 18:45:01.648141 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.648119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5" event={"ID":"224064a5-060e-4ff9-bf02-cc1b10971d35","Type":"ContainerStarted","Data":"eaa65ebec182dd827f3b9272edbc4925f9dbaf0b12777caf5f5c539f069c7ba1"}
Sep 16 18:45:01.649552 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.649528    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x" event={"ID":"a0b4b104-0a71-41bd-91b1-8cdceda29a8b","Type":"ContainerStarted","Data":"44c7923b7556f7643f001e613fe4ce52310b3093730700ca1bad1e33675a1906"}
Sep 16 18:45:01.650802 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.650782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7" event={"ID":"130a948c-0e38-4cae-bf51-c10b332bf1a5","Type":"ContainerStarted","Data":"bf13246317654f6ae9a7981a22f2b0df086eb92b1a8936598c04510fc1730d60"}
Sep 16 18:45:01.688098 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.685059    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq" podStartSLOduration=1.685037065 podStartE2EDuration="1.685037065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.667192341 +0000 UTC m=+1044.376431733" watchObservedRunningTime="2026-09-16 18:45:01.685037065 +0000 UTC m=+1044.394276459"
Sep 16 18:45:01.688098 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.686472    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x" podStartSLOduration=1.686452859 podStartE2EDuration="1.686452859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.684499842 +0000 UTC m=+1044.393739238" watchObservedRunningTime="2026-09-16 18:45:01.686452859 +0000 UTC m=+1044.395692252"
Sep 16 18:45:01.701198 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.701141    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7" podStartSLOduration=1.7011224889999998 podStartE2EDuration="1.701122489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.700598478 +0000 UTC m=+1044.409837871" watchObservedRunningTime="2026-09-16 18:45:01.701122489 +0000 UTC m=+1044.410361872"
Sep 16 18:45:01.719048 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.719001    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp" podStartSLOduration=1.7189835759999998 podStartE2EDuration="1.718983576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.716986878 +0000 UTC m=+1044.426226271" watchObservedRunningTime="2026-09-16 18:45:01.718983576 +0000 UTC m=+1044.428222964"
Sep 16 18:45:01.734462 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:01.734414    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5" podStartSLOduration=1.734399657 podStartE2EDuration="1.734399657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.733152734 +0000 UTC m=+1044.442392126" watchObservedRunningTime="2026-09-16 18:45:01.734399657 +0000 UTC m=+1044.443639073"
Sep 16 18:45:02.657854 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:02.657816    2580 generic.go:356] "Generic (PLEG): container finished" podID="224064a5-060e-4ff9-bf02-cc1b10971d35" containerID="eaa65ebec182dd827f3b9272edbc4925f9dbaf0b12777caf5f5c539f069c7ba1" exitCode=0
Sep 16 18:45:02.658363 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:02.657901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5" event={"ID":"224064a5-060e-4ff9-bf02-cc1b10971d35","Type":"ContainerDied","Data":"eaa65ebec182dd827f3b9272edbc4925f9dbaf0b12777caf5f5c539f069c7ba1"}
Sep 16 18:45:02.659852 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:02.659455    2580 generic.go:356] "Generic (PLEG): container finished" podID="a0b4b104-0a71-41bd-91b1-8cdceda29a8b" containerID="44c7923b7556f7643f001e613fe4ce52310b3093730700ca1bad1e33675a1906" exitCode=0
Sep 16 18:45:02.659852 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:02.659550    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x" event={"ID":"a0b4b104-0a71-41bd-91b1-8cdceda29a8b","Type":"ContainerDied","Data":"44c7923b7556f7643f001e613fe4ce52310b3093730700ca1bad1e33675a1906"}
Sep 16 18:45:02.661697 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:02.661664    2580 generic.go:356] "Generic (PLEG): container finished" podID="130a948c-0e38-4cae-bf51-c10b332bf1a5" containerID="bf13246317654f6ae9a7981a22f2b0df086eb92b1a8936598c04510fc1730d60" exitCode=0
Sep 16 18:45:02.661851 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:02.661828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7" event={"ID":"130a948c-0e38-4cae-bf51-c10b332bf1a5","Type":"ContainerDied","Data":"bf13246317654f6ae9a7981a22f2b0df086eb92b1a8936598c04510fc1730d60"}
Sep 16 18:45:02.663995 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:02.663964    2580 generic.go:356] "Generic (PLEG): container finished" podID="5c0df27b-d20a-4d5b-843a-327145560f55" containerID="cde9d301ef5ebea3005175b7ad9a8629144d3fcd1f85b1915e21550e1836323e" exitCode=0
Sep 16 18:45:02.664734 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:02.664708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp" event={"ID":"5c0df27b-d20a-4d5b-843a-327145560f55","Type":"ContainerDied","Data":"cde9d301ef5ebea3005175b7ad9a8629144d3fcd1f85b1915e21550e1836323e"}
Sep 16 18:45:03.892349 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:03.891485    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7"
Sep 16 18:45:03.931087 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:03.930971    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp"
Sep 16 18:45:03.937871 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:03.937846    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5"
Sep 16 18:45:03.941670 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:03.941604    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x"
Sep 16 18:45:04.682310 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.682263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5" event={"ID":"224064a5-060e-4ff9-bf02-cc1b10971d35","Type":"ContainerDied","Data":"b8d38dd31c5c893066bb46b388fd2810b8541204bd1c10d874b829a97d028bd7"}
Sep 16 18:45:04.682535 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.682389    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8d38dd31c5c893066bb46b388fd2810b8541204bd1c10d874b829a97d028bd7"
Sep 16 18:45:04.682535 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.682469    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-8lcv5"
Sep 16 18:45:04.688107 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.688074    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x" event={"ID":"a0b4b104-0a71-41bd-91b1-8cdceda29a8b","Type":"ContainerDied","Data":"2118ca50f744f26804fbeed4b2b38d28ec9151d020a3d7bfe21f208fca8d8254"}
Sep 16 18:45:04.688227 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.688127    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2118ca50f744f26804fbeed4b2b38d28ec9151d020a3d7bfe21f208fca8d8254"
Sep 16 18:45:04.688227 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.688082    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x"
Sep 16 18:45:04.691448 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.689848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7" event={"ID":"130a948c-0e38-4cae-bf51-c10b332bf1a5","Type":"ContainerDied","Data":"fb87b834450f94e3fce683369380a8bf020c371712e8de188bd1bb165b8e4e18"}
Sep 16 18:45:04.691448 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.689890    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb87b834450f94e3fce683369380a8bf020c371712e8de188bd1bb165b8e4e18"
Sep 16 18:45:04.691448 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.689947    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7"
Sep 16 18:45:04.693596 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.693576    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp"
Sep 16 18:45:04.693596 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.693597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp" event={"ID":"5c0df27b-d20a-4d5b-843a-327145560f55","Type":"ContainerDied","Data":"e78951a2d04b6f994f3809bd36cf5fc20b2eb2d576cee50b914cbf471bd50599"}
Sep 16 18:45:04.693772 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:04.693636    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e78951a2d04b6f994f3809bd36cf5fc20b2eb2d576cee50b914cbf471bd50599"
Sep 16 18:45:10.435003 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:10.434968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:45:10.730445 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:10.730357    2580 generic.go:356] "Generic (PLEG): container finished" podID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerID="412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb" exitCode=0
Sep 16 18:45:10.730587 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:10.730368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" event={"ID":"93b77c1a-cc53-4a16-b737-abbe7eec3103","Type":"ContainerDied","Data":"412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb"}
Sep 16 18:45:10.730842 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:10.730828    2580 scope.go:117] "RemoveContainer" containerID="412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb"
Sep 16 18:45:10.731939 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:10.731905    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:45:10.732036 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:10.731973    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:45:10.732036 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:10.732007    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:45:10.783978 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:10.783950    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:10.784117 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:10.784004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:10.799080 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:10.799052    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:11.738502 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:11.738464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" event={"ID":"93b77c1a-cc53-4a16-b737-abbe7eec3103","Type":"ContainerStarted","Data":"12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6"}
Sep 16 18:45:11.750259 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:11.750227    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:45:12.502889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:12.502851    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:45:12.504043 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:12.504013    2580 patch_prober.go:28] interesting pod/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:45:12.504177 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:12.504084    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:45:12.528596 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:12.528571    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:45:13.502397 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:13.502362    2580 patch_prober.go:28] interesting pod/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:45:13.502801 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:13.502418    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:45:14.502317 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:14.502282    2580 patch_prober.go:28] interesting pod/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:45:14.502513 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:14.502363    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:45:15.502546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:15.502512    2580 patch_prober.go:28] interesting pod/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:45:15.502955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:15.502567    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:45:16.502947 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:16.502906    2580 patch_prober.go:28] interesting pod/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:45:16.503461 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:16.502977    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:45:17.507839 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:17.507793    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:45:17.509050 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:17.509016    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:45:30.052300 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.052214    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"]
Sep 16 18:45:30.052808 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.052786    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130a948c-0e38-4cae-bf51-c10b332bf1a5" containerName="cleanup"
Sep 16 18:45:30.052895 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.052812    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="130a948c-0e38-4cae-bf51-c10b332bf1a5" containerName="cleanup"
Sep 16 18:45:30.052895 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.052839    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0b4b104-0a71-41bd-91b1-8cdceda29a8b" containerName="cleanup"
Sep 16 18:45:30.052895 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.052847    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0b4b104-0a71-41bd-91b1-8cdceda29a8b" containerName="cleanup"
Sep 16 18:45:30.052895 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.052875    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c0df27b-d20a-4d5b-843a-327145560f55" containerName="cleanup"
Sep 16 18:45:30.052895 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.052884    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c0df27b-d20a-4d5b-843a-327145560f55" containerName="cleanup"
Sep 16 18:45:30.053131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.052898    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224064a5-060e-4ff9-bf02-cc1b10971d35" containerName="cleanup"
Sep 16 18:45:30.053131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.052907    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="224064a5-060e-4ff9-bf02-cc1b10971d35" containerName="cleanup"
Sep 16 18:45:30.053131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.053004    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="224064a5-060e-4ff9-bf02-cc1b10971d35" containerName="cleanup"
Sep 16 18:45:30.053131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.053018    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="130a948c-0e38-4cae-bf51-c10b332bf1a5" containerName="cleanup"
Sep 16 18:45:30.053131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.053028    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0b4b104-0a71-41bd-91b1-8cdceda29a8b" containerName="cleanup"
Sep 16 18:45:30.053131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.053040    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c0df27b-d20a-4d5b-843a-327145560f55" containerName="cleanup"
Sep 16 18:45:30.056450 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.056430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:45:30.058946 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.058924    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6498ead1-dockercfg-jtxzp\""
Sep 16 18:45:30.059079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.058956    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6498ead1\""
Sep 16 18:45:30.067400 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.067374    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"]
Sep 16 18:45:30.211571 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.211534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422a8972-af85-4e63-8d95-03eb2bf74f89-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld\" (UID: \"422a8972-af85-4e63-8d95-03eb2bf74f89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:45:30.211743 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.211594    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qx5\" (UniqueName: \"kubernetes.io/projected/422a8972-af85-4e63-8d95-03eb2bf74f89-kube-api-access-m5qx5\") pod \"payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld\" (UID: \"422a8972-af85-4e63-8d95-03eb2bf74f89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:45:30.313186 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.313101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qx5\" (UniqueName: \"kubernetes.io/projected/422a8972-af85-4e63-8d95-03eb2bf74f89-kube-api-access-m5qx5\") pod \"payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld\" (UID: \"422a8972-af85-4e63-8d95-03eb2bf74f89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:45:30.313370 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.313212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422a8972-af85-4e63-8d95-03eb2bf74f89-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld\" (UID: \"422a8972-af85-4e63-8d95-03eb2bf74f89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:45:30.313897 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.313877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422a8972-af85-4e63-8d95-03eb2bf74f89-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld\" (UID: \"422a8972-af85-4e63-8d95-03eb2bf74f89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:45:30.325156 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.325128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qx5\" (UniqueName: \"kubernetes.io/projected/422a8972-af85-4e63-8d95-03eb2bf74f89-kube-api-access-m5qx5\") pod \"payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld\" (UID: \"422a8972-af85-4e63-8d95-03eb2bf74f89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:45:30.368212 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.368167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:45:30.639947 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.639922    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"]
Sep 16 18:45:30.642289 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:45:30.642255    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422a8972_af85_4e63_8d95_03eb2bf74f89.slice/crio-5d9fdc977bb8d5090da9c8a33477b032de139d805bc052c1636016fc13f37423 WatchSource:0}: Error finding container 5d9fdc977bb8d5090da9c8a33477b032de139d805bc052c1636016fc13f37423: Status 404 returned error can't find the container with id 5d9fdc977bb8d5090da9c8a33477b032de139d805bc052c1636016fc13f37423
Sep 16 18:45:30.846465 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.846433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld" event={"ID":"422a8972-af85-4e63-8d95-03eb2bf74f89","Type":"ContainerStarted","Data":"3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9"}
Sep 16 18:45:30.846628 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.846496    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:45:30.846628 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.846519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld" event={"ID":"422a8972-af85-4e63-8d95-03eb2bf74f89","Type":"ContainerStarted","Data":"5d9fdc977bb8d5090da9c8a33477b032de139d805bc052c1636016fc13f37423"}
Sep 16 18:45:30.879975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:30.879922    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld" podStartSLOduration=0.879903415 podStartE2EDuration="879.903415ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:30.878261725 +0000 UTC m=+1073.587501117" watchObservedRunningTime="2026-09-16 18:45:30.879903415 +0000 UTC m=+1073.589142808"
Sep 16 18:45:38.955952 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:38.955904    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:45:41.853757 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:41.853726    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:45:53.756730 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:53.756693    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"]
Sep 16 18:45:53.760572 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:53.760543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"
Sep 16 18:45:53.763311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:53.763290    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:45:53.763509 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:53.763493    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:45:53.773024 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:53.772992    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"]
Sep 16 18:45:53.844127 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:53.844097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e4a88d-4b28-4b8d-9d77-28fc97a8467d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5\" (UID: \"93e4a88d-4b28-4b8d-9d77-28fc97a8467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"
Sep 16 18:45:53.945710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:53.945677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e4a88d-4b28-4b8d-9d77-28fc97a8467d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5\" (UID: \"93e4a88d-4b28-4b8d-9d77-28fc97a8467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"
Sep 16 18:45:53.946283 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:53.946265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e4a88d-4b28-4b8d-9d77-28fc97a8467d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5\" (UID: \"93e4a88d-4b28-4b8d-9d77-28fc97a8467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"
Sep 16 18:45:54.073187 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:54.073158    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"
Sep 16 18:45:54.351094 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:54.351072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"]
Sep 16 18:45:54.353523 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:45:54.353489    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e4a88d_4b28_4b8d_9d77_28fc97a8467d.slice/crio-7b2654c7ebb4eb493db103efba6168e0ffd9c53325ac1b0b317d2a093821e02a WatchSource:0}: Error finding container 7b2654c7ebb4eb493db103efba6168e0ffd9c53325ac1b0b317d2a093821e02a: Status 404 returned error can't find the container with id 7b2654c7ebb4eb493db103efba6168e0ffd9c53325ac1b0b317d2a093821e02a
Sep 16 18:45:54.985548 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:54.985512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5" event={"ID":"93e4a88d-4b28-4b8d-9d77-28fc97a8467d","Type":"ContainerStarted","Data":"1104dcb279baf9caa68bd800fc737e6c9447a9f18622007d04e5a0a2aa602a15"}
Sep 16 18:45:54.985955 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:54.985581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5" event={"ID":"93e4a88d-4b28-4b8d-9d77-28fc97a8467d","Type":"ContainerStarted","Data":"7b2654c7ebb4eb493db103efba6168e0ffd9c53325ac1b0b317d2a093821e02a"}
Sep 16 18:45:55.003470 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:55.003427    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5" podStartSLOduration=2.003413249 podStartE2EDuration="2.003413249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:55.002058653 +0000 UTC m=+1097.711298045" watchObservedRunningTime="2026-09-16 18:45:55.003413249 +0000 UTC m=+1097.712652638"
Sep 16 18:45:55.991923 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:55.991888    2580 generic.go:356] "Generic (PLEG): container finished" podID="93e4a88d-4b28-4b8d-9d77-28fc97a8467d" containerID="1104dcb279baf9caa68bd800fc737e6c9447a9f18622007d04e5a0a2aa602a15" exitCode=0
Sep 16 18:45:55.992309 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:55.991977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5" event={"ID":"93e4a88d-4b28-4b8d-9d77-28fc97a8467d","Type":"ContainerDied","Data":"1104dcb279baf9caa68bd800fc737e6c9447a9f18622007d04e5a0a2aa602a15"}
Sep 16 18:45:57.147822 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:57.147787    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"
Sep 16 18:45:57.283400 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:57.283371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e4a88d-4b28-4b8d-9d77-28fc97a8467d-maas-api-service-ca\") pod \"93e4a88d-4b28-4b8d-9d77-28fc97a8467d\" (UID: \"93e4a88d-4b28-4b8d-9d77-28fc97a8467d\") "
Sep 16 18:45:57.283733 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:57.283707    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e4a88d-4b28-4b8d-9d77-28fc97a8467d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93e4a88d-4b28-4b8d-9d77-28fc97a8467d" (UID: "93e4a88d-4b28-4b8d-9d77-28fc97a8467d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:57.384833 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:57.384798    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e4a88d-4b28-4b8d-9d77-28fc97a8467d-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:45:58.005268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:58.005235    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"
Sep 16 18:45:58.005467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:58.005282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5" event={"ID":"93e4a88d-4b28-4b8d-9d77-28fc97a8467d","Type":"ContainerDied","Data":"7b2654c7ebb4eb493db103efba6168e0ffd9c53325ac1b0b317d2a093821e02a"}
Sep 16 18:45:58.005467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:45:58.005369    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b2654c7ebb4eb493db103efba6168e0ffd9c53325ac1b0b317d2a093821e02a"
Sep 16 18:46:07.139801 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:07.139762    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:46:08.780534 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:08.780499    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"]
Sep 16 18:46:08.780997 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:08.780730    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld" podUID="422a8972-af85-4e63-8d95-03eb2bf74f89" containerName="payload-pre-processing" containerID="cri-o://3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9" gracePeriod=30
Sep 16 18:46:08.966101 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:08.966069    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"]
Sep 16 18:46:08.970269 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:08.970245    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6498ead1-fkgd5"]
Sep 16 18:46:09.037001 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.036915    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:46:09.074316 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.074289    2580 generic.go:356] "Generic (PLEG): container finished" podID="422a8972-af85-4e63-8d95-03eb2bf74f89" containerID="3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9" exitCode=0
Sep 16 18:46:09.074494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.074346    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"
Sep 16 18:46:09.074494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.074375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld" event={"ID":"422a8972-af85-4e63-8d95-03eb2bf74f89","Type":"ContainerDied","Data":"3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9"}
Sep 16 18:46:09.074494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.074436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld" event={"ID":"422a8972-af85-4e63-8d95-03eb2bf74f89","Type":"ContainerDied","Data":"5d9fdc977bb8d5090da9c8a33477b032de139d805bc052c1636016fc13f37423"}
Sep 16 18:46:09.074494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.074469    2580 scope.go:117] "RemoveContainer" containerID="3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9"
Sep 16 18:46:09.085574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.085556    2580 scope.go:117] "RemoveContainer" containerID="3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9"
Sep 16 18:46:09.085853 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:46:09.085826    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9\": container with ID starting with 3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9 not found: ID does not exist" containerID="3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9"
Sep 16 18:46:09.085919 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.085862    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9"} err="failed to get container status \"3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9\": rpc error: code = NotFound desc = could not find container \"3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9\": container with ID starting with 3c5df9f2e3b8bfdea91be723a83e98644a8cb09f5aee1377f694d540d49143e9 not found: ID does not exist"
Sep 16 18:46:09.199443 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.199410    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5qx5\" (UniqueName: \"kubernetes.io/projected/422a8972-af85-4e63-8d95-03eb2bf74f89-kube-api-access-m5qx5\") pod \"422a8972-af85-4e63-8d95-03eb2bf74f89\" (UID: \"422a8972-af85-4e63-8d95-03eb2bf74f89\") "
Sep 16 18:46:09.199597 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.199503    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422a8972-af85-4e63-8d95-03eb2bf74f89-plugins-config-volume\") pod \"422a8972-af85-4e63-8d95-03eb2bf74f89\" (UID: \"422a8972-af85-4e63-8d95-03eb2bf74f89\") "
Sep 16 18:46:09.199864 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.199836    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422a8972-af85-4e63-8d95-03eb2bf74f89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "422a8972-af85-4e63-8d95-03eb2bf74f89" (UID: "422a8972-af85-4e63-8d95-03eb2bf74f89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:09.201609 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.201584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422a8972-af85-4e63-8d95-03eb2bf74f89-kube-api-access-m5qx5" (OuterVolumeSpecName: "kube-api-access-m5qx5") pod "422a8972-af85-4e63-8d95-03eb2bf74f89" (UID: "422a8972-af85-4e63-8d95-03eb2bf74f89"). InnerVolumeSpecName "kube-api-access-m5qx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:09.300166 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.300102    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422a8972-af85-4e63-8d95-03eb2bf74f89-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.300166 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.300125    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qx5\" (UniqueName: \"kubernetes.io/projected/422a8972-af85-4e63-8d95-03eb2bf74f89-kube-api-access-m5qx5\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.398226 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.398186    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"]
Sep 16 18:46:09.400922 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.400899    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6498ead1-5dd75db98-vc4ld"]
Sep 16 18:46:09.787628 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.787592    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="422a8972-af85-4e63-8d95-03eb2bf74f89" path="/var/lib/kubelet/pods/422a8972-af85-4e63-8d95-03eb2bf74f89/volumes"
Sep 16 18:46:09.788009 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:09.787976    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93e4a88d-4b28-4b8d-9d77-28fc97a8467d" path="/var/lib/kubelet/pods/93e4a88d-4b28-4b8d-9d77-28fc97a8467d/volumes"
Sep 16 18:46:40.723788 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.723750    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"]
Sep 16 18:46:40.724235 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.724173    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422a8972-af85-4e63-8d95-03eb2bf74f89" containerName="payload-pre-processing"
Sep 16 18:46:40.724235 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.724186    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="422a8972-af85-4e63-8d95-03eb2bf74f89" containerName="payload-pre-processing"
Sep 16 18:46:40.724235 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.724196    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e4a88d-4b28-4b8d-9d77-28fc97a8467d" containerName="revoke-keys"
Sep 16 18:46:40.724235 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.724202    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e4a88d-4b28-4b8d-9d77-28fc97a8467d" containerName="revoke-keys"
Sep 16 18:46:40.724412 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.724272    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="422a8972-af85-4e63-8d95-03eb2bf74f89" containerName="payload-pre-processing"
Sep 16 18:46:40.724412 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.724285    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e4a88d-4b28-4b8d-9d77-28fc97a8467d" containerName="revoke-keys"
Sep 16 18:46:40.728738 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.728720    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:46:40.731921 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.731894    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ee98a5\""
Sep 16 18:46:40.732155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.732135    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ee98a5-dockercfg-79hhh\""
Sep 16 18:46:40.743071 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.743047    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"]
Sep 16 18:46:40.750467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.750444    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"]
Sep 16 18:46:40.753887 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.753867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:46:40.771651 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.771627    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"]
Sep 16 18:46:40.813883 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.813845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e45844-e9ee-4733-94b8-a91ad94a643d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr\" (UID: \"80e45844-e9ee-4733-94b8-a91ad94a643d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:46:40.814045 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.813905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2swgw\" (UniqueName: \"kubernetes.io/projected/80e45844-e9ee-4733-94b8-a91ad94a643d-kube-api-access-2swgw\") pod \"payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr\" (UID: \"80e45844-e9ee-4733-94b8-a91ad94a643d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:46:40.914515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.914480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2swgw\" (UniqueName: \"kubernetes.io/projected/80e45844-e9ee-4733-94b8-a91ad94a643d-kube-api-access-2swgw\") pod \"payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr\" (UID: \"80e45844-e9ee-4733-94b8-a91ad94a643d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:46:40.914515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.914518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6\" (UID: \"eb8f4b0f-35e5-4346-823d-90b1c71e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:46:40.914718 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.914660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4w9s\" (UniqueName: \"kubernetes.io/projected/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-kube-api-access-x4w9s\") pod \"payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6\" (UID: \"eb8f4b0f-35e5-4346-823d-90b1c71e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:46:40.914888 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.914856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e45844-e9ee-4733-94b8-a91ad94a643d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr\" (UID: \"80e45844-e9ee-4733-94b8-a91ad94a643d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:46:40.915521 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.915502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e45844-e9ee-4733-94b8-a91ad94a643d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr\" (UID: \"80e45844-e9ee-4733-94b8-a91ad94a643d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:46:40.924804 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:40.924775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2swgw\" (UniqueName: \"kubernetes.io/projected/80e45844-e9ee-4733-94b8-a91ad94a643d-kube-api-access-2swgw\") pod \"payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr\" (UID: \"80e45844-e9ee-4733-94b8-a91ad94a643d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:46:41.016181 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:41.016103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6\" (UID: \"eb8f4b0f-35e5-4346-823d-90b1c71e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:46:41.016181 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:41.016144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4w9s\" (UniqueName: \"kubernetes.io/projected/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-kube-api-access-x4w9s\") pod \"payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6\" (UID: \"eb8f4b0f-35e5-4346-823d-90b1c71e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:46:41.016819 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:41.016800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6\" (UID: \"eb8f4b0f-35e5-4346-823d-90b1c71e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:46:41.025544 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:41.025523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4w9s\" (UniqueName: \"kubernetes.io/projected/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-kube-api-access-x4w9s\") pod \"payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6\" (UID: \"eb8f4b0f-35e5-4346-823d-90b1c71e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:46:41.039441 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:41.039420    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:46:41.063633 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:41.063608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:46:41.231482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:41.231440    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:46:41.321836 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:41.321809    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"]
Sep 16 18:46:41.323649 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:46:41.323626    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e45844_e9ee_4733_94b8_a91ad94a643d.slice/crio-3e1403d9b47748805832b9492603ce051ef4336b7675462b6e8fd1fb688e27e0 WatchSource:0}: Error finding container 3e1403d9b47748805832b9492603ce051ef4336b7675462b6e8fd1fb688e27e0: Status 404 returned error can't find the container with id 3e1403d9b47748805832b9492603ce051ef4336b7675462b6e8fd1fb688e27e0
Sep 16 18:46:41.355165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:41.355142    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"]
Sep 16 18:46:41.360662 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:46:41.360633    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8f4b0f_35e5_4346_823d_90b1c71e16ac.slice/crio-33ebf21b00ceb2e1eb9a08e6ca9832604ab08c16420be9a00b6aeac4cde7166e WatchSource:0}: Error finding container 33ebf21b00ceb2e1eb9a08e6ca9832604ab08c16420be9a00b6aeac4cde7166e: Status 404 returned error can't find the container with id 33ebf21b00ceb2e1eb9a08e6ca9832604ab08c16420be9a00b6aeac4cde7166e
Sep 16 18:46:42.269842 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:42.269801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr" event={"ID":"80e45844-e9ee-4733-94b8-a91ad94a643d","Type":"ContainerStarted","Data":"e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c"}
Sep 16 18:46:42.270300 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:42.269863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr" event={"ID":"80e45844-e9ee-4733-94b8-a91ad94a643d","Type":"ContainerStarted","Data":"3e1403d9b47748805832b9492603ce051ef4336b7675462b6e8fd1fb688e27e0"}
Sep 16 18:46:42.270300 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:42.269895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:46:42.271566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:42.271540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6" event={"ID":"eb8f4b0f-35e5-4346-823d-90b1c71e16ac","Type":"ContainerStarted","Data":"4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884"}
Sep 16 18:46:42.271681 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:42.271588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6" event={"ID":"eb8f4b0f-35e5-4346-823d-90b1c71e16ac","Type":"ContainerStarted","Data":"33ebf21b00ceb2e1eb9a08e6ca9832604ab08c16420be9a00b6aeac4cde7166e"}
Sep 16 18:46:42.271681 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:42.271613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:46:42.292248 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:42.292193    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr" podStartSLOduration=2.292179183 podStartE2EDuration="2.292179183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:42.289393907 +0000 UTC m=+1144.998633299" watchObservedRunningTime="2026-09-16 18:46:42.292179183 +0000 UTC m=+1145.001418575"
Sep 16 18:46:42.307038 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:42.306991    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6" podStartSLOduration=2.306972154 podStartE2EDuration="2.306972154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:42.306572894 +0000 UTC m=+1145.015812286" watchObservedRunningTime="2026-09-16 18:46:42.306972154 +0000 UTC m=+1145.016211549"
Sep 16 18:46:52.183152 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:52.183118    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"]
Sep 16 18:46:52.187063 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:52.187041    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"
Sep 16 18:46:52.189701 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:52.189677    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:46:52.189816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:52.189683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:46:52.196318 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:52.196296    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"]
Sep 16 18:46:52.226583 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:52.226555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a6de83-8f41-46e6-8e6a-b095d2b58b9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq\" (UID: \"82a6de83-8f41-46e6-8e6a-b095d2b58b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"
Sep 16 18:46:52.327589 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:52.327556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a6de83-8f41-46e6-8e6a-b095d2b58b9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq\" (UID: \"82a6de83-8f41-46e6-8e6a-b095d2b58b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"
Sep 16 18:46:52.328235 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:52.328210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a6de83-8f41-46e6-8e6a-b095d2b58b9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq\" (UID: \"82a6de83-8f41-46e6-8e6a-b095d2b58b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"
Sep 16 18:46:52.499212 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:52.499119    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"
Sep 16 18:46:52.772571 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:52.772539    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"]
Sep 16 18:46:52.774295 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:46:52.774266    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a6de83_8f41_46e6_8e6a_b095d2b58b9a.slice/crio-7384737452c31b6019f26d31bdb845c873691d144338c29e33ca9555a568cd89 WatchSource:0}: Error finding container 7384737452c31b6019f26d31bdb845c873691d144338c29e33ca9555a568cd89: Status 404 returned error can't find the container with id 7384737452c31b6019f26d31bdb845c873691d144338c29e33ca9555a568cd89
Sep 16 18:46:53.282422 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:53.282389    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:46:53.282895 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:53.282463    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:46:53.339584 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:53.339493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq" event={"ID":"82a6de83-8f41-46e6-8e6a-b095d2b58b9a","Type":"ContainerStarted","Data":"5c2f4aea64793250e61e674971c7d5c592451a82dfdd99c52e654bb673b70d1d"}
Sep 16 18:46:53.339584 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:53.339559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq" event={"ID":"82a6de83-8f41-46e6-8e6a-b095d2b58b9a","Type":"ContainerStarted","Data":"7384737452c31b6019f26d31bdb845c873691d144338c29e33ca9555a568cd89"}
Sep 16 18:46:53.362456 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:53.362370    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq" podStartSLOduration=1.362345556 podStartE2EDuration="1.362345556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:53.360683309 +0000 UTC m=+1156.069922701" watchObservedRunningTime="2026-09-16 18:46:53.362345556 +0000 UTC m=+1156.071584948"
Sep 16 18:46:54.346864 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:54.346823    2580 generic.go:356] "Generic (PLEG): container finished" podID="82a6de83-8f41-46e6-8e6a-b095d2b58b9a" containerID="5c2f4aea64793250e61e674971c7d5c592451a82dfdd99c52e654bb673b70d1d" exitCode=0
Sep 16 18:46:54.347316 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:54.346903    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq" event={"ID":"82a6de83-8f41-46e6-8e6a-b095d2b58b9a","Type":"ContainerDied","Data":"5c2f4aea64793250e61e674971c7d5c592451a82dfdd99c52e654bb673b70d1d"}
Sep 16 18:46:55.505190 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.505166    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"
Sep 16 18:46:55.559290 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.559261    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a6de83-8f41-46e6-8e6a-b095d2b58b9a-maas-api-service-ca\") pod \"82a6de83-8f41-46e6-8e6a-b095d2b58b9a\" (UID: \"82a6de83-8f41-46e6-8e6a-b095d2b58b9a\") "
Sep 16 18:46:55.559684 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.559663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a6de83-8f41-46e6-8e6a-b095d2b58b9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82a6de83-8f41-46e6-8e6a-b095d2b58b9a" (UID: "82a6de83-8f41-46e6-8e6a-b095d2b58b9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:55.660359 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.660255    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a6de83-8f41-46e6-8e6a-b095d2b58b9a-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:46:55.887782 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.887749    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"]
Sep 16 18:46:55.888198 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.888183    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a6de83-8f41-46e6-8e6a-b095d2b58b9a" containerName="revoke-keys"
Sep 16 18:46:55.888244 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.888199    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a6de83-8f41-46e6-8e6a-b095d2b58b9a" containerName="revoke-keys"
Sep 16 18:46:55.888279 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.888272    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a6de83-8f41-46e6-8e6a-b095d2b58b9a" containerName="revoke-keys"
Sep 16 18:46:55.891406 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.891385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:55.894292 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.894264    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-8737d2-openshift-default-dockercfg-hsq9w\""
Sep 16 18:46:55.905614 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.905594    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"]
Sep 16 18:46:55.964325 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.964228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-socket\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:55.964325 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.964273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-envoy\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:55.964325 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.964311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-certs\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:55.964591 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.964368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-data\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:55.964591 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.964481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb167c06-6fa4-4437-9994-709b682d6634-istio-podinfo\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:55.964591 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.964522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wtd\" (UniqueName: \"kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-kube-api-access-h8wtd\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:55.964705 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.964597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-istio-token\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:55.964705 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.964663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-credential-socket\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:55.964705 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:55.964682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb167c06-6fa4-4437-9994-709b682d6634-istiod-ca-cert\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.065297 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.065256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb167c06-6fa4-4437-9994-709b682d6634-istio-podinfo\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.065496 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.065314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wtd\" (UniqueName: \"kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-kube-api-access-h8wtd\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.065496 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.065381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-istio-token\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.065496 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.065416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-credential-socket\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.065496 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.065446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb167c06-6fa4-4437-9994-709b682d6634-istiod-ca-cert\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.065751 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.065518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-socket\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.065751 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.065545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-envoy\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.065751 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.065582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-certs\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.065751 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.065608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-data\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.065977 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.065833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-credential-socket\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.066107 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.066075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-certs\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.066203 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.066152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-socket\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.066403 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.066382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb167c06-6fa4-4437-9994-709b682d6634-istiod-ca-cert\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.066520 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.066502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-data\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.067866 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.067837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-envoy\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.068112 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.068094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb167c06-6fa4-4437-9994-709b682d6634-istio-podinfo\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.076726 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.076696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-istio-token\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.076837 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.076790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wtd\" (UniqueName: \"kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-kube-api-access-h8wtd\") pod \"e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") " pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.201544 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.201506    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:56.359770 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.359739    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"
Sep 16 18:46:56.359940 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.359789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq" event={"ID":"82a6de83-8f41-46e6-8e6a-b095d2b58b9a","Type":"ContainerDied","Data":"7384737452c31b6019f26d31bdb845c873691d144338c29e33ca9555a568cd89"}
Sep 16 18:46:56.359940 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.359841    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7384737452c31b6019f26d31bdb845c873691d144338c29e33ca9555a568cd89"
Sep 16 18:46:56.485953 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.485926    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"]
Sep 16 18:46:56.487701 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:46:56.487669    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb167c06_6fa4_4437_9994_709b682d6634.slice/crio-709b6d0c377a2729f681089d28a26a97e619144f7d896af39ff08a2d5d0f8a28 WatchSource:0}: Error finding container 709b6d0c377a2729f681089d28a26a97e619144f7d896af39ff08a2d5d0f8a28: Status 404 returned error can't find the container with id 709b6d0c377a2729f681089d28a26a97e619144f7d896af39ff08a2d5d0f8a28
Sep 16 18:46:56.490192 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.490158    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:46:56.490303 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.490236    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:46:56.490303 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:56.490282    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:46:57.370047 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:57.370003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" event={"ID":"cb167c06-6fa4-4437-9994-709b682d6634","Type":"ContainerStarted","Data":"55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0"}
Sep 16 18:46:57.370459 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:57.370123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" event={"ID":"cb167c06-6fa4-4437-9994-709b682d6634","Type":"ContainerStarted","Data":"709b6d0c377a2729f681089d28a26a97e619144f7d896af39ff08a2d5d0f8a28"}
Sep 16 18:46:57.398211 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:57.398146    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" podStartSLOduration=2.3981244139999998 podStartE2EDuration="2.398124414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:57.397588268 +0000 UTC m=+1160.106827656" watchObservedRunningTime="2026-09-16 18:46:57.398124414 +0000 UTC m=+1160.107363808"
Sep 16 18:46:58.202429 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:58.202381    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:58.207602 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:58.207578    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:58.379997 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:58.379961    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:46:58.381201 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:46:58.381176    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:47:05.980043 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:05.979957    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"]
Sep 16 18:47:05.983606 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:05.983583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:47:05.986765 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:05.986733    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-8737d2\""
Sep 16 18:47:05.986765 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:05.986733    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-8737d2-dockercfg-7sq6m\""
Sep 16 18:47:05.998982 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:05.998958    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"]
Sep 16 18:47:06.070717 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:06.070682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96644388-feb0-4638-9fdb-4df53d0d523a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6\" (UID: \"96644388-feb0-4638-9fdb-4df53d0d523a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:47:06.070938 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:06.070748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v95t\" (UniqueName: \"kubernetes.io/projected/96644388-feb0-4638-9fdb-4df53d0d523a-kube-api-access-5v95t\") pod \"payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6\" (UID: \"96644388-feb0-4638-9fdb-4df53d0d523a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:47:06.172276 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:06.172230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96644388-feb0-4638-9fdb-4df53d0d523a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6\" (UID: \"96644388-feb0-4638-9fdb-4df53d0d523a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:47:06.172463 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:06.172306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v95t\" (UniqueName: \"kubernetes.io/projected/96644388-feb0-4638-9fdb-4df53d0d523a-kube-api-access-5v95t\") pod \"payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6\" (UID: \"96644388-feb0-4638-9fdb-4df53d0d523a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:47:06.173006 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:06.172986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96644388-feb0-4638-9fdb-4df53d0d523a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6\" (UID: \"96644388-feb0-4638-9fdb-4df53d0d523a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:47:06.181384 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:06.181360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v95t\" (UniqueName: \"kubernetes.io/projected/96644388-feb0-4638-9fdb-4df53d0d523a-kube-api-access-5v95t\") pod \"payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6\" (UID: \"96644388-feb0-4638-9fdb-4df53d0d523a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:47:06.293816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:06.293786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:47:06.567842 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:06.567814    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"]
Sep 16 18:47:06.569310 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:47:06.569280    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96644388_feb0_4638_9fdb_4df53d0d523a.slice/crio-72ce99f8d89c7f61e825f5e79706eb1e1f27a66a82030ebd0a06b94ad4f27dca WatchSource:0}: Error finding container 72ce99f8d89c7f61e825f5e79706eb1e1f27a66a82030ebd0a06b94ad4f27dca: Status 404 returned error can't find the container with id 72ce99f8d89c7f61e825f5e79706eb1e1f27a66a82030ebd0a06b94ad4f27dca
Sep 16 18:47:07.430007 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:07.429974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6" event={"ID":"96644388-feb0-4638-9fdb-4df53d0d523a","Type":"ContainerStarted","Data":"748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa"}
Sep 16 18:47:07.430414 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:07.430035    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:47:07.430414 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:07.430060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6" event={"ID":"96644388-feb0-4638-9fdb-4df53d0d523a","Type":"ContainerStarted","Data":"72ce99f8d89c7f61e825f5e79706eb1e1f27a66a82030ebd0a06b94ad4f27dca"}
Sep 16 18:47:07.450482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:07.450441    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6" podStartSLOduration=2.450424258 podStartE2EDuration="2.450424258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:07.448530202 +0000 UTC m=+1170.157769594" watchObservedRunningTime="2026-09-16 18:47:07.450424258 +0000 UTC m=+1170.159663650"
Sep 16 18:47:08.952417 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:08.952385    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x"]
Sep 16 18:47:08.959301 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:08.959274    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-16ae98-29826405-jvk6x"]
Sep 16 18:47:08.968905 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:08.968870    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:47:09.025311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.025273    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"]
Sep 16 18:47:09.025677 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.025625    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj" podUID="2dbf790b-2c71-4728-a6ad-0aededbf37b8" containerName="payload-processing" containerID="cri-o://72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60" gracePeriod=30
Sep 16 18:47:09.030081 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.030056    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"]
Sep 16 18:47:09.030529 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.030259    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246" podUID="95ebdf24-000f-4272-b5bb-d02cb2086c4f" containerName="payload-pre-processing" containerID="cri-o://0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661" gracePeriod=30
Sep 16 18:47:09.293169 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.293139    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"]
Sep 16 18:47:09.296866 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.296843    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-16ae98-cxhkq"]
Sep 16 18:47:09.313530 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.313512    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:47:09.316271 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.316254    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:47:09.406952 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.406922    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v2j2\" (UniqueName: \"kubernetes.io/projected/95ebdf24-000f-4272-b5bb-d02cb2086c4f-kube-api-access-5v2j2\") pod \"95ebdf24-000f-4272-b5bb-d02cb2086c4f\" (UID: \"95ebdf24-000f-4272-b5bb-d02cb2086c4f\") "
Sep 16 18:47:09.407149 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.406960    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vlpc\" (UniqueName: \"kubernetes.io/projected/2dbf790b-2c71-4728-a6ad-0aededbf37b8-kube-api-access-7vlpc\") pod \"2dbf790b-2c71-4728-a6ad-0aededbf37b8\" (UID: \"2dbf790b-2c71-4728-a6ad-0aededbf37b8\") "
Sep 16 18:47:09.407149 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.407100    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ebdf24-000f-4272-b5bb-d02cb2086c4f-plugins-config-volume\") pod \"95ebdf24-000f-4272-b5bb-d02cb2086c4f\" (UID: \"95ebdf24-000f-4272-b5bb-d02cb2086c4f\") "
Sep 16 18:47:09.407149 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.407127    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbf790b-2c71-4728-a6ad-0aededbf37b8-plugins-config-volume\") pod \"2dbf790b-2c71-4728-a6ad-0aededbf37b8\" (UID: \"2dbf790b-2c71-4728-a6ad-0aededbf37b8\") "
Sep 16 18:47:09.407522 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.407497    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ebdf24-000f-4272-b5bb-d02cb2086c4f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95ebdf24-000f-4272-b5bb-d02cb2086c4f" (UID: "95ebdf24-000f-4272-b5bb-d02cb2086c4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:09.407628 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.407527    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dbf790b-2c71-4728-a6ad-0aededbf37b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2dbf790b-2c71-4728-a6ad-0aededbf37b8" (UID: "2dbf790b-2c71-4728-a6ad-0aededbf37b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:09.409680 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.409657    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ebdf24-000f-4272-b5bb-d02cb2086c4f-kube-api-access-5v2j2" (OuterVolumeSpecName: "kube-api-access-5v2j2") pod "95ebdf24-000f-4272-b5bb-d02cb2086c4f" (UID: "95ebdf24-000f-4272-b5bb-d02cb2086c4f"). InnerVolumeSpecName "kube-api-access-5v2j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:09.409792 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.409685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbf790b-2c71-4728-a6ad-0aededbf37b8-kube-api-access-7vlpc" (OuterVolumeSpecName: "kube-api-access-7vlpc") pod "2dbf790b-2c71-4728-a6ad-0aededbf37b8" (UID: "2dbf790b-2c71-4728-a6ad-0aededbf37b8"). InnerVolumeSpecName "kube-api-access-7vlpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:09.442790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.442763    2580 generic.go:356] "Generic (PLEG): container finished" podID="95ebdf24-000f-4272-b5bb-d02cb2086c4f" containerID="0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661" exitCode=0
Sep 16 18:47:09.442899 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.442807    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"
Sep 16 18:47:09.442899 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.442844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246" event={"ID":"95ebdf24-000f-4272-b5bb-d02cb2086c4f","Type":"ContainerDied","Data":"0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661"}
Sep 16 18:47:09.443005 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.442901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246" event={"ID":"95ebdf24-000f-4272-b5bb-d02cb2086c4f","Type":"ContainerDied","Data":"23bc6864fc952c913804616ca990d1478d0930d0bd583837482a881e2ab94171"}
Sep 16 18:47:09.443005 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.442937    2580 scope.go:117] "RemoveContainer" containerID="0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661"
Sep 16 18:47:09.444260 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.444241    2580 generic.go:356] "Generic (PLEG): container finished" podID="2dbf790b-2c71-4728-a6ad-0aededbf37b8" containerID="72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60" exitCode=0
Sep 16 18:47:09.444369 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.444283    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"
Sep 16 18:47:09.444369 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.444313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj" event={"ID":"2dbf790b-2c71-4728-a6ad-0aededbf37b8","Type":"ContainerDied","Data":"72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60"}
Sep 16 18:47:09.444466 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.444389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj" event={"ID":"2dbf790b-2c71-4728-a6ad-0aededbf37b8","Type":"ContainerDied","Data":"6088cb215950ee4ce50c49f8f788d1f5ff06468bb82859c310daf1333def8015"}
Sep 16 18:47:09.452149 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.452130    2580 scope.go:117] "RemoveContainer" containerID="0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661"
Sep 16 18:47:09.452486 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:09.452464    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661\": container with ID starting with 0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661 not found: ID does not exist" containerID="0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661"
Sep 16 18:47:09.452578 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.452493    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661"} err="failed to get container status \"0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661\": rpc error: code = NotFound desc = could not find container \"0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661\": container with ID starting with 0cb7979d33933413e5ecf1028fe607e5514c3cdc7e97027b09d821e7565cb661 not found: ID does not exist"
Sep 16 18:47:09.452578 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.452509    2580 scope.go:117] "RemoveContainer" containerID="72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60"
Sep 16 18:47:09.463702 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.463686    2580 scope.go:117] "RemoveContainer" containerID="72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60"
Sep 16 18:47:09.464012 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:09.463994    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60\": container with ID starting with 72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60 not found: ID does not exist" containerID="72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60"
Sep 16 18:47:09.464081 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.464021    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60"} err="failed to get container status \"72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60\": rpc error: code = NotFound desc = could not find container \"72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60\": container with ID starting with 72b701f32f8cc55e4f729dd9550be5fa890640cee69b92542a6228d2787adf60 not found: ID does not exist"
Sep 16 18:47:09.480493 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.480461    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"]
Sep 16 18:47:09.490550 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.490498    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-16ae98-8b4d759db-mx246"]
Sep 16 18:47:09.508506 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.508484    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ebdf24-000f-4272-b5bb-d02cb2086c4f-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:09.508506 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.508504    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbf790b-2c71-4728-a6ad-0aededbf37b8-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:09.508612 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.508515    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v2j2\" (UniqueName: \"kubernetes.io/projected/95ebdf24-000f-4272-b5bb-d02cb2086c4f-kube-api-access-5v2j2\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:09.508612 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.508523    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vlpc\" (UniqueName: \"kubernetes.io/projected/2dbf790b-2c71-4728-a6ad-0aededbf37b8-kube-api-access-7vlpc\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:09.526301 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.526278    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"]
Sep 16 18:47:09.533743 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.533722    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-16ae98-65747c8f97-dwvfj"]
Sep 16 18:47:09.785971 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.785927    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbf790b-2c71-4728-a6ad-0aededbf37b8" path="/var/lib/kubelet/pods/2dbf790b-2c71-4728-a6ad-0aededbf37b8/volumes"
Sep 16 18:47:09.786364 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.786324    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a6de83-8f41-46e6-8e6a-b095d2b58b9a" path="/var/lib/kubelet/pods/82a6de83-8f41-46e6-8e6a-b095d2b58b9a/volumes"
Sep 16 18:47:09.786800 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.786775    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95ebdf24-000f-4272-b5bb-d02cb2086c4f" path="/var/lib/kubelet/pods/95ebdf24-000f-4272-b5bb-d02cb2086c4f/volumes"
Sep 16 18:47:09.787108 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:09.787092    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0b4b104-0a71-41bd-91b1-8cdceda29a8b" path="/var/lib/kubelet/pods/a0b4b104-0a71-41bd-91b1-8cdceda29a8b/volumes"
Sep 16 18:47:15.244578 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.244546    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7636-afef80"]
Sep 16 18:47:15.245034 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.244917    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95ebdf24-000f-4272-b5bb-d02cb2086c4f" containerName="payload-pre-processing"
Sep 16 18:47:15.245034 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.244928    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="95ebdf24-000f-4272-b5bb-d02cb2086c4f" containerName="payload-pre-processing"
Sep 16 18:47:15.245034 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.244939    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbf790b-2c71-4728-a6ad-0aededbf37b8" containerName="payload-processing"
Sep 16 18:47:15.245034 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.244945    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbf790b-2c71-4728-a6ad-0aededbf37b8" containerName="payload-processing"
Sep 16 18:47:15.245034 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.245022    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbf790b-2c71-4728-a6ad-0aededbf37b8" containerName="payload-processing"
Sep 16 18:47:15.245034 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.245033    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="95ebdf24-000f-4272-b5bb-d02cb2086c4f" containerName="payload-pre-processing"
Sep 16 18:47:15.249695 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.249677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-afef80"
Sep 16 18:47:15.255049 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.255028    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4g8kx\""
Sep 16 18:47:15.257642 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.257622    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-afef80"]
Sep 16 18:47:15.369572 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.369537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ftc\" (UniqueName: \"kubernetes.io/projected/200c99ae-3beb-4956-b81f-880e722db04c-kube-api-access-s2ftc\") pod \"test-curl-7636-afef80\" (UID: \"200c99ae-3beb-4956-b81f-880e722db04c\") " pod="opendatahub/test-curl-7636-afef80"
Sep 16 18:47:15.470204 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.470165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ftc\" (UniqueName: \"kubernetes.io/projected/200c99ae-3beb-4956-b81f-880e722db04c-kube-api-access-s2ftc\") pod \"test-curl-7636-afef80\" (UID: \"200c99ae-3beb-4956-b81f-880e722db04c\") " pod="opendatahub/test-curl-7636-afef80"
Sep 16 18:47:15.481184 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.481152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ftc\" (UniqueName: \"kubernetes.io/projected/200c99ae-3beb-4956-b81f-880e722db04c-kube-api-access-s2ftc\") pod \"test-curl-7636-afef80\" (UID: \"200c99ae-3beb-4956-b81f-880e722db04c\") " pod="opendatahub/test-curl-7636-afef80"
Sep 16 18:47:15.559805 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.559775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-afef80"
Sep 16 18:47:15.866882 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:15.866850    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-afef80"]
Sep 16 18:47:15.867400 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:47:15.867369    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod200c99ae_3beb_4956_b81f_880e722db04c.slice/crio-7ebe356c2f8b96a3840cebe7491674bfc345ff07c1988336212d6c07b45b18f5 WatchSource:0}: Error finding container 7ebe356c2f8b96a3840cebe7491674bfc345ff07c1988336212d6c07b45b18f5: Status 404 returned error can't find the container with id 7ebe356c2f8b96a3840cebe7491674bfc345ff07c1988336212d6c07b45b18f5
Sep 16 18:47:16.491172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:16.491137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-afef80" event={"ID":"200c99ae-3beb-4956-b81f-880e722db04c","Type":"ContainerStarted","Data":"ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d"}
Sep 16 18:47:16.491584 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:16.491186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-afef80" event={"ID":"200c99ae-3beb-4956-b81f-880e722db04c","Type":"ContainerStarted","Data":"7ebe356c2f8b96a3840cebe7491674bfc345ff07c1988336212d6c07b45b18f5"}
Sep 16 18:47:16.511967 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:16.511921    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7636-afef80" podStartSLOduration=1.5119077669999998 podStartE2EDuration="1.511907767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:16.510219566 +0000 UTC m=+1179.219458964" watchObservedRunningTime="2026-09-16 18:47:16.511907767 +0000 UTC m=+1179.221147182"
Sep 16 18:47:17.155755 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.150572    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7636-afef80"]
Sep 16 18:47:17.162850 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.162821    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7636-afef80"]
Sep 16 18:47:17.283103 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.283067    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7636-a77fe0"]
Sep 16 18:47:17.283531 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.283517    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="200c99ae-3beb-4956-b81f-880e722db04c" containerName="test-curl-7636-afef80"
Sep 16 18:47:17.283584 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.283533    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="200c99ae-3beb-4956-b81f-880e722db04c" containerName="test-curl-7636-afef80"
Sep 16 18:47:17.283620 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.283594    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="200c99ae-3beb-4956-b81f-880e722db04c" containerName="test-curl-7636-afef80"
Sep 16 18:47:17.286702 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.286679    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-a77fe0"
Sep 16 18:47:17.293268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.293244    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-a77fe0"]
Sep 16 18:47:17.387904 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.387864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ql7l\" (UniqueName: \"kubernetes.io/projected/0b483fbc-197d-421d-99a1-bc28f50512f8-kube-api-access-6ql7l\") pod \"test-curl-7636-a77fe0\" (UID: \"0b483fbc-197d-421d-99a1-bc28f50512f8\") " pod="opendatahub/test-curl-7636-a77fe0"
Sep 16 18:47:17.488864 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.488832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ql7l\" (UniqueName: \"kubernetes.io/projected/0b483fbc-197d-421d-99a1-bc28f50512f8-kube-api-access-6ql7l\") pod \"test-curl-7636-a77fe0\" (UID: \"0b483fbc-197d-421d-99a1-bc28f50512f8\") " pod="opendatahub/test-curl-7636-a77fe0"
Sep 16 18:47:17.498060 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:17.498034    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7636-afef80" container="test-curl-7636-afef80"
Sep 16 18:47:17.499076 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.499040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ql7l\" (UniqueName: \"kubernetes.io/projected/0b483fbc-197d-421d-99a1-bc28f50512f8-kube-api-access-6ql7l\") pod \"test-curl-7636-a77fe0\" (UID: \"0b483fbc-197d-421d-99a1-bc28f50512f8\") " pod="opendatahub/test-curl-7636-a77fe0"
Sep 16 18:47:17.596725 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.596640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-a77fe0"
Sep 16 18:47:17.876732 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:17.876708    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-a77fe0"]
Sep 16 18:47:17.879255 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:47:17.879219    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b483fbc_197d_421d_99a1_bc28f50512f8.slice/crio-ce655e8e47864f10b50fd5e09ac2a7e874537bdfc417cdd4983434a05125af4c WatchSource:0}: Error finding container ce655e8e47864f10b50fd5e09ac2a7e874537bdfc417cdd4983434a05125af4c: Status 404 returned error can't find the container with id ce655e8e47864f10b50fd5e09ac2a7e874537bdfc417cdd4983434a05125af4c
Sep 16 18:47:18.437831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:18.437803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:47:18.506378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:18.506314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-a77fe0" event={"ID":"0b483fbc-197d-421d-99a1-bc28f50512f8","Type":"ContainerStarted","Data":"4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed"}
Sep 16 18:47:18.506822 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:18.506393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-a77fe0" event={"ID":"0b483fbc-197d-421d-99a1-bc28f50512f8","Type":"ContainerStarted","Data":"ce655e8e47864f10b50fd5e09ac2a7e874537bdfc417cdd4983434a05125af4c"}
Sep 16 18:47:18.506822 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:18.506432    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7636-afef80" podUID="200c99ae-3beb-4956-b81f-880e722db04c" containerName="test-curl-7636-afef80" containerID="cri-o://ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d" gracePeriod=2
Sep 16 18:47:18.537495 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:18.537449    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7636-a77fe0" podStartSLOduration=1.537433653 podStartE2EDuration="1.537433653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:18.535563855 +0000 UTC m=+1181.244803247" watchObservedRunningTime="2026-09-16 18:47:18.537433653 +0000 UTC m=+1181.246673115"
Sep 16 18:47:19.204778 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.204743    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7636-a77fe0"]
Sep 16 18:47:19.212407 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.212370    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7636-a77fe0"]
Sep 16 18:47:19.331887 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.331857    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7636-92670e"]
Sep 16 18:47:19.332296 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.332281    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b483fbc-197d-421d-99a1-bc28f50512f8" containerName="test-curl-7636-a77fe0"
Sep 16 18:47:19.332364 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.332297    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b483fbc-197d-421d-99a1-bc28f50512f8" containerName="test-curl-7636-a77fe0"
Sep 16 18:47:19.332449 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.332391    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b483fbc-197d-421d-99a1-bc28f50512f8" containerName="test-curl-7636-a77fe0"
Sep 16 18:47:19.335519 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.335504    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-92670e"
Sep 16 18:47:19.343716 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.343693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-92670e"]
Sep 16 18:47:19.406419 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.406385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfrfh\" (UniqueName: \"kubernetes.io/projected/b39863cd-f61d-487d-aac9-a2e66218d341-kube-api-access-qfrfh\") pod \"test-curl-7636-92670e\" (UID: \"b39863cd-f61d-487d-aac9-a2e66218d341\") " pod="opendatahub/test-curl-7636-92670e"
Sep 16 18:47:19.507844 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.507753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfrfh\" (UniqueName: \"kubernetes.io/projected/b39863cd-f61d-487d-aac9-a2e66218d341-kube-api-access-qfrfh\") pod \"test-curl-7636-92670e\" (UID: \"b39863cd-f61d-487d-aac9-a2e66218d341\") " pod="opendatahub/test-curl-7636-92670e"
Sep 16 18:47:19.512620 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:19.512591    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7636-a77fe0" container="test-curl-7636-a77fe0"
Sep 16 18:47:19.516546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.516526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfrfh\" (UniqueName: \"kubernetes.io/projected/b39863cd-f61d-487d-aac9-a2e66218d341-kube-api-access-qfrfh\") pod \"test-curl-7636-92670e\" (UID: \"b39863cd-f61d-487d-aac9-a2e66218d341\") " pod="opendatahub/test-curl-7636-92670e"
Sep 16 18:47:19.645079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.645048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-92670e"
Sep 16 18:47:19.913257 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:19.913234    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-92670e"]
Sep 16 18:47:19.915035 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:47:19.915007    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb39863cd_f61d_487d_aac9_a2e66218d341.slice/crio-7b4abb63bec81031a2325a75c6574897a68dd9a3c0f96bb53bd1cab40234fe53 WatchSource:0}: Error finding container 7b4abb63bec81031a2325a75c6574897a68dd9a3c0f96bb53bd1cab40234fe53: Status 404 returned error can't find the container with id 7b4abb63bec81031a2325a75c6574897a68dd9a3c0f96bb53bd1cab40234fe53
Sep 16 18:47:20.519593 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:20.519558    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-92670e" event={"ID":"b39863cd-f61d-487d-aac9-a2e66218d341","Type":"ContainerStarted","Data":"395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86"}
Sep 16 18:47:20.520229 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:20.519612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-92670e" event={"ID":"b39863cd-f61d-487d-aac9-a2e66218d341","Type":"ContainerStarted","Data":"7b4abb63bec81031a2325a75c6574897a68dd9a3c0f96bb53bd1cab40234fe53"}
Sep 16 18:47:20.520229 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:20.519686    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7636-a77fe0" podUID="0b483fbc-197d-421d-99a1-bc28f50512f8" containerName="test-curl-7636-a77fe0" containerID="cri-o://4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed" gracePeriod=2
Sep 16 18:47:20.540343 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:20.540282    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7636-92670e" podStartSLOduration=1.540266279 podStartE2EDuration="1.540266279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:20.539487053 +0000 UTC m=+1183.248726447" watchObservedRunningTime="2026-09-16 18:47:20.540266279 +0000 UTC m=+1183.249505670"
Sep 16 18:47:20.678954 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:20.678930    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-afef80"
Sep 16 18:47:20.719087 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:20.719053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2ftc\" (UniqueName: \"kubernetes.io/projected/200c99ae-3beb-4956-b81f-880e722db04c-kube-api-access-s2ftc\") pod \"200c99ae-3beb-4956-b81f-880e722db04c\" (UID: \"200c99ae-3beb-4956-b81f-880e722db04c\") "
Sep 16 18:47:20.721141 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:20.721116    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200c99ae-3beb-4956-b81f-880e722db04c-kube-api-access-s2ftc" (OuterVolumeSpecName: "kube-api-access-s2ftc") pod "200c99ae-3beb-4956-b81f-880e722db04c" (UID: "200c99ae-3beb-4956-b81f-880e722db04c"). InnerVolumeSpecName "kube-api-access-s2ftc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:20.820647 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:20.820606    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2ftc\" (UniqueName: \"kubernetes.io/projected/200c99ae-3beb-4956-b81f-880e722db04c-kube-api-access-s2ftc\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:21.179236 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.179149    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7636-92670e"]
Sep 16 18:47:21.182987 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.182962    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7636-92670e"]
Sep 16 18:47:21.307048 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.307017    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7636-e218ba"]
Sep 16 18:47:21.307442 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.307427    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b39863cd-f61d-487d-aac9-a2e66218d341" containerName="test-curl-7636-92670e"
Sep 16 18:47:21.307491 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.307443    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b39863cd-f61d-487d-aac9-a2e66218d341" containerName="test-curl-7636-92670e"
Sep 16 18:47:21.307548 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.307540    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b39863cd-f61d-487d-aac9-a2e66218d341" containerName="test-curl-7636-92670e"
Sep 16 18:47:21.310566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.310549    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-e218ba"
Sep 16 18:47:21.324419 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.324394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-e218ba"]
Sep 16 18:47:21.326605 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.326575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcqq8\" (UniqueName: \"kubernetes.io/projected/977d4d70-5b77-4c01-a202-f4c8509519bb-kube-api-access-hcqq8\") pod \"test-curl-7636-e218ba\" (UID: \"977d4d70-5b77-4c01-a202-f4c8509519bb\") " pod="opendatahub/test-curl-7636-e218ba"
Sep 16 18:47:21.427508 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.427479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcqq8\" (UniqueName: \"kubernetes.io/projected/977d4d70-5b77-4c01-a202-f4c8509519bb-kube-api-access-hcqq8\") pod \"test-curl-7636-e218ba\" (UID: \"977d4d70-5b77-4c01-a202-f4c8509519bb\") " pod="opendatahub/test-curl-7636-e218ba"
Sep 16 18:47:21.440211 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.440150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcqq8\" (UniqueName: \"kubernetes.io/projected/977d4d70-5b77-4c01-a202-f4c8509519bb-kube-api-access-hcqq8\") pod \"test-curl-7636-e218ba\" (UID: \"977d4d70-5b77-4c01-a202-f4c8509519bb\") " pod="opendatahub/test-curl-7636-e218ba"
Sep 16 18:47:21.526264 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.526231    2580 generic.go:356] "Generic (PLEG): container finished" podID="200c99ae-3beb-4956-b81f-880e722db04c" containerID="ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d" exitCode=137
Sep 16 18:47:21.526704 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.526275    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-afef80"
Sep 16 18:47:21.526704 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.526359    2580 scope.go:117] "RemoveContainer" containerID="ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d"
Sep 16 18:47:21.526704 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:21.526559    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7636-92670e" container="test-curl-7636-92670e"
Sep 16 18:47:21.536477 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.536456    2580 scope.go:117] "RemoveContainer" containerID="ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d"
Sep 16 18:47:21.536771 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:21.536751    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d\": container with ID starting with ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d not found: ID does not exist" containerID="ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d"
Sep 16 18:47:21.536841 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.536784    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d"} err="failed to get container status \"ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d\": rpc error: code = NotFound desc = could not find container \"ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d\": container with ID starting with ecf5bb862ae12dbf16a20cd6c8e2129d18af41e1278903be1bb5af6d126afc1d not found: ID does not exist"
Sep 16 18:47:21.538642 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:21.538613    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7636-afef80\" is forbidden: User \"system:node:ip-10-0-141-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-16.ec2.internal' and this object" podUID="200c99ae-3beb-4956-b81f-880e722db04c" pod="opendatahub/test-curl-7636-afef80"
Sep 16 18:47:21.619918 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.619889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-e218ba"
Sep 16 18:47:21.784232 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.784190    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="200c99ae-3beb-4956-b81f-880e722db04c" path="/var/lib/kubelet/pods/200c99ae-3beb-4956-b81f-880e722db04c/volumes"
Sep 16 18:47:21.904776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:21.904747    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-e218ba"]
Sep 16 18:47:21.907685 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:47:21.907647    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod977d4d70_5b77_4c01_a202_f4c8509519bb.slice/crio-2631590b3d94479858fde86a2d514814498b80ab401e9bd9b009d6473ca8c071 WatchSource:0}: Error finding container 2631590b3d94479858fde86a2d514814498b80ab401e9bd9b009d6473ca8c071: Status 404 returned error can't find the container with id 2631590b3d94479858fde86a2d514814498b80ab401e9bd9b009d6473ca8c071
Sep 16 18:47:22.537791 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:22.537762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-e218ba" event={"ID":"977d4d70-5b77-4c01-a202-f4c8509519bb","Type":"ContainerStarted","Data":"bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da"}
Sep 16 18:47:22.538132 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:22.537810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-e218ba" event={"ID":"977d4d70-5b77-4c01-a202-f4c8509519bb","Type":"ContainerStarted","Data":"2631590b3d94479858fde86a2d514814498b80ab401e9bd9b009d6473ca8c071"}
Sep 16 18:47:22.538132 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:22.537936    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7636-92670e" podUID="b39863cd-f61d-487d-aac9-a2e66218d341" containerName="test-curl-7636-92670e" containerID="cri-o://395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86" gracePeriod=2
Sep 16 18:47:22.559340 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:22.559261    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7636-e218ba" podStartSLOduration=1.5592412420000001 podStartE2EDuration="1.559241242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:22.557104711 +0000 UTC m=+1185.266344108" watchObservedRunningTime="2026-09-16 18:47:22.559241242 +0000 UTC m=+1185.268480639"
Sep 16 18:47:22.679624 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:22.679592    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-a77fe0"
Sep 16 18:47:22.740385 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:22.740355    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ql7l\" (UniqueName: \"kubernetes.io/projected/0b483fbc-197d-421d-99a1-bc28f50512f8-kube-api-access-6ql7l\") pod \"0b483fbc-197d-421d-99a1-bc28f50512f8\" (UID: \"0b483fbc-197d-421d-99a1-bc28f50512f8\") "
Sep 16 18:47:22.742550 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:22.742516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b483fbc-197d-421d-99a1-bc28f50512f8-kube-api-access-6ql7l" (OuterVolumeSpecName: "kube-api-access-6ql7l") pod "0b483fbc-197d-421d-99a1-bc28f50512f8" (UID: "0b483fbc-197d-421d-99a1-bc28f50512f8"). InnerVolumeSpecName "kube-api-access-6ql7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:22.841465 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:22.841378    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ql7l\" (UniqueName: \"kubernetes.io/projected/0b483fbc-197d-421d-99a1-bc28f50512f8-kube-api-access-6ql7l\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.174831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:23.174738    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7636-e218ba"]
Sep 16 18:47:23.183404 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:23.183372    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7636-e218ba"]
Sep 16 18:47:23.544168 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:23.544129    2580 generic.go:356] "Generic (PLEG): container finished" podID="0b483fbc-197d-421d-99a1-bc28f50512f8" containerID="4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed" exitCode=137
Sep 16 18:47:23.544588 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:23.544178    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-a77fe0"
Sep 16 18:47:23.544588 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:23.544218    2580 scope.go:117] "RemoveContainer" containerID="4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed"
Sep 16 18:47:23.544588 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:23.544497    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7636-e218ba" container="test-curl-7636-e218ba"
Sep 16 18:47:23.553340 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:23.553312    2580 scope.go:117] "RemoveContainer" containerID="4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed"
Sep 16 18:47:23.553603 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:23.553583    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed\": container with ID starting with 4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed not found: ID does not exist" containerID="4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed"
Sep 16 18:47:23.553679 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:23.553612    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed"} err="failed to get container status \"4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed\": rpc error: code = NotFound desc = could not find container \"4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed\": container with ID starting with 4311f2e8f82d8b7a11baca56fa2b9d065bc331158a656acf1b7406f0ca14daed not found: ID does not exist"
Sep 16 18:47:23.554916 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:23.554895    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7636-a77fe0\" is forbidden: User \"system:node:ip-10-0-141-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-16.ec2.internal' and this object" podUID="0b483fbc-197d-421d-99a1-bc28f50512f8" pod="opendatahub/test-curl-7636-a77fe0"
Sep 16 18:47:23.784529 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:23.784497    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b483fbc-197d-421d-99a1-bc28f50512f8" path="/var/lib/kubelet/pods/0b483fbc-197d-421d-99a1-bc28f50512f8/volumes"
Sep 16 18:47:24.145918 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.145576    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"]
Sep 16 18:47:24.146160 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.146139    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977d4d70-5b77-4c01-a202-f4c8509519bb" containerName="test-curl-7636-e218ba"
Sep 16 18:47:24.146245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.146164    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="977d4d70-5b77-4c01-a202-f4c8509519bb" containerName="test-curl-7636-e218ba"
Sep 16 18:47:24.146304 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.146261    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="977d4d70-5b77-4c01-a202-f4c8509519bb" containerName="test-curl-7636-e218ba"
Sep 16 18:47:24.150967 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.150941    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.153790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.153770    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d91d95b4-openshift-default-dockercfg-cv4kd\""
Sep 16 18:47:24.173754 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.173677    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"]
Sep 16 18:47:24.254839 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.254800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-token\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.254839 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.254843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-podinfo\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.255032 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.254875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-credential-socket\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.255032 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.254891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8mw9\" (UniqueName: \"kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-kube-api-access-l8mw9\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.255032 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.254997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-envoy\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.255128 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.255059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-certs\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.255128 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.255092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-socket\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.255196 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.255179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-data\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.255231 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.255217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istiod-ca-cert\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.355789 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.355746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-data\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.355940 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.355868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istiod-ca-cert\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.355940 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.355892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-token\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.355940 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.355910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-podinfo\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.356141 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.356066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-credential-socket\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.356141 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.356115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-data\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.356254 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.356215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8mw9\" (UniqueName: \"kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-kube-api-access-l8mw9\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.356354 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.356319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-envoy\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.356433 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.356387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-credential-socket\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.356433 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.356404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-certs\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.356716 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.356688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istiod-ca-cert\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.356830 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.356720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-certs\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.356830 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.356438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-socket\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.356830 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.356763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-socket\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.358505 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.358486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-envoy\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.358747 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.358725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-podinfo\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.368361 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.367818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-token\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.368361 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.368031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8mw9\" (UniqueName: \"kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-kube-api-access-l8mw9\") pod \"e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") " pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.471751 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.471650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:24.552532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.552500    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7636-e218ba" podUID="977d4d70-5b77-4c01-a202-f4c8509519bb" containerName="test-curl-7636-e218ba" containerID="cri-o://bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da" gracePeriod=2
Sep 16 18:47:24.741118 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.741093    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-92670e"
Sep 16 18:47:24.756208 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.755766    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"]
Sep 16 18:47:24.758718 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:47:24.758687    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb74c06_dff3_4c09_a7e0_cf2a86e74937.slice/crio-f09cadf3977f642a970354d72dc02bf1605ce5b95a9e4f5a25b1d9c1b598bd2d WatchSource:0}: Error finding container f09cadf3977f642a970354d72dc02bf1605ce5b95a9e4f5a25b1d9c1b598bd2d: Status 404 returned error can't find the container with id f09cadf3977f642a970354d72dc02bf1605ce5b95a9e4f5a25b1d9c1b598bd2d
Sep 16 18:47:24.784353 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.782680    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfrfh\" (UniqueName: \"kubernetes.io/projected/b39863cd-f61d-487d-aac9-a2e66218d341-kube-api-access-qfrfh\") pod \"b39863cd-f61d-487d-aac9-a2e66218d341\" (UID: \"b39863cd-f61d-487d-aac9-a2e66218d341\") "
Sep 16 18:47:24.788494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.785829    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:24.788494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.785915    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:24.788494 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.785955    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:24.790874 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.790837    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39863cd-f61d-487d-aac9-a2e66218d341-kube-api-access-qfrfh" (OuterVolumeSpecName: "kube-api-access-qfrfh") pod "b39863cd-f61d-487d-aac9-a2e66218d341" (UID: "b39863cd-f61d-487d-aac9-a2e66218d341"). InnerVolumeSpecName "kube-api-access-qfrfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:24.884709 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:24.884667    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfrfh\" (UniqueName: \"kubernetes.io/projected/b39863cd-f61d-487d-aac9-a2e66218d341-kube-api-access-qfrfh\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:25.561509 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:25.560259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" event={"ID":"7eb74c06-dff3-4c09-a7e0-cf2a86e74937","Type":"ContainerStarted","Data":"82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8"}
Sep 16 18:47:25.561509 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:25.560339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" event={"ID":"7eb74c06-dff3-4c09-a7e0-cf2a86e74937","Type":"ContainerStarted","Data":"f09cadf3977f642a970354d72dc02bf1605ce5b95a9e4f5a25b1d9c1b598bd2d"}
Sep 16 18:47:25.566348 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:25.563408    2580 generic.go:356] "Generic (PLEG): container finished" podID="b39863cd-f61d-487d-aac9-a2e66218d341" containerID="395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86" exitCode=137
Sep 16 18:47:25.566348 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:25.563481    2580 scope.go:117] "RemoveContainer" containerID="395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86"
Sep 16 18:47:25.566348 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:25.563611    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-92670e"
Sep 16 18:47:25.587548 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:25.587179    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" podStartSLOduration=1.587157403 podStartE2EDuration="1.587157403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:25.585679574 +0000 UTC m=+1188.294918989" watchObservedRunningTime="2026-09-16 18:47:25.587157403 +0000 UTC m=+1188.296396800"
Sep 16 18:47:25.593504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:25.593480    2580 scope.go:117] "RemoveContainer" containerID="395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86"
Sep 16 18:47:25.593746 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:25.593714    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7636-92670e\" is forbidden: User \"system:node:ip-10-0-141-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-16.ec2.internal' and this object" podUID="b39863cd-f61d-487d-aac9-a2e66218d341" pod="opendatahub/test-curl-7636-92670e"
Sep 16 18:47:25.594151 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:25.594102    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86\": container with ID starting with 395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86 not found: ID does not exist" containerID="395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86"
Sep 16 18:47:25.594312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:25.594278    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86"} err="failed to get container status \"395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86\": rpc error: code = NotFound desc = could not find container \"395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86\": container with ID starting with 395b4df439e15a28432f22cbc457caac520009791a33e44ccaf0239d164c5d86 not found: ID does not exist"
Sep 16 18:47:25.787740 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:25.787702    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b39863cd-f61d-487d-aac9-a2e66218d341" path="/var/lib/kubelet/pods/b39863cd-f61d-487d-aac9-a2e66218d341/volumes"
Sep 16 18:47:26.472531 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:26.472497    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:26.481904 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:26.481878    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:26.574428 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:26.574389    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:26.578802 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:26.578779    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:26.775778 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:26.775637    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-e218ba"
Sep 16 18:47:26.906833 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:26.906794    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcqq8\" (UniqueName: \"kubernetes.io/projected/977d4d70-5b77-4c01-a202-f4c8509519bb-kube-api-access-hcqq8\") pod \"977d4d70-5b77-4c01-a202-f4c8509519bb\" (UID: \"977d4d70-5b77-4c01-a202-f4c8509519bb\") "
Sep 16 18:47:26.908938 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:26.908910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977d4d70-5b77-4c01-a202-f4c8509519bb-kube-api-access-hcqq8" (OuterVolumeSpecName: "kube-api-access-hcqq8") pod "977d4d70-5b77-4c01-a202-f4c8509519bb" (UID: "977d4d70-5b77-4c01-a202-f4c8509519bb"). InnerVolumeSpecName "kube-api-access-hcqq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:27.007831 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:27.007749    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcqq8\" (UniqueName: \"kubernetes.io/projected/977d4d70-5b77-4c01-a202-f4c8509519bb-kube-api-access-hcqq8\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:27.588385 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:27.584779    2580 generic.go:356] "Generic (PLEG): container finished" podID="977d4d70-5b77-4c01-a202-f4c8509519bb" containerID="bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da" exitCode=137
Sep 16 18:47:27.588385 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:27.585694    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-e218ba"
Sep 16 18:47:27.589484 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:27.589460    2580 scope.go:117] "RemoveContainer" containerID="bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da"
Sep 16 18:47:27.598248 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:27.598219    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7636-e218ba\" is forbidden: User \"system:node:ip-10-0-141-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-16.ec2.internal' and this object" podUID="977d4d70-5b77-4c01-a202-f4c8509519bb" pod="opendatahub/test-curl-7636-e218ba"
Sep 16 18:47:27.599621 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:27.599605    2580 scope.go:117] "RemoveContainer" containerID="bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da"
Sep 16 18:47:27.599949 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:27.599928    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da\": container with ID starting with bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da not found: ID does not exist" containerID="bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da"
Sep 16 18:47:27.599999 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:27.599957    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da"} err="failed to get container status \"bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da\": rpc error: code = NotFound desc = could not find container \"bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da\": container with ID starting with bc5ba01224cc551537de6a3df1c70db5fbb3af40bc50d1be9ebb6131005515da not found: ID does not exist"
Sep 16 18:47:27.787453 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:27.787416    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="977d4d70-5b77-4c01-a202-f4c8509519bb" path="/var/lib/kubelet/pods/977d4d70-5b77-4c01-a202-f4c8509519bb/volumes"
Sep 16 18:47:27.789069 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:27.789042    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7636-e218ba\" is forbidden: User \"system:node:ip-10-0-141-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-16.ec2.internal' and this object" podUID="977d4d70-5b77-4c01-a202-f4c8509519bb" pod="opendatahub/test-curl-7636-e218ba"
Sep 16 18:47:33.623846 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:33.623810    2580 generic.go:356] "Generic (PLEG): container finished" podID="cb167c06-6fa4-4437-9994-709b682d6634" containerID="55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0" exitCode=0
Sep 16 18:47:33.624403 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:33.623886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" event={"ID":"cb167c06-6fa4-4437-9994-709b682d6634","Type":"ContainerDied","Data":"55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0"}
Sep 16 18:47:33.624477 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:33.624452    2580 scope.go:117] "RemoveContainer" containerID="55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0"
Sep 16 18:47:33.625681 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:33.625650    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:33.625772 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:33.625720    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:33.625772 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:33.625755    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:34.631140 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:34.631106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" event={"ID":"cb167c06-6fa4-4437-9994-709b682d6634","Type":"ContainerStarted","Data":"70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61"}
Sep 16 18:47:35.202570 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:35.202536    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:47:35.202776 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:35.202756    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 16 18:47:35.202821 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:35.202802    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 16 18:47:36.202131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:36.202093    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 16 18:47:36.202564 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:36.202154    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 16 18:47:37.202232 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:37.202201    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 16 18:47:37.202625 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:37.202252    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 16 18:47:37.817223 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:37.817194    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:47:37.818482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:37.818459    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:47:37.889454 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:37.889416    2580 scope.go:117] "RemoveContainer" containerID="fd3c06726d42aaebb084b53ea147bcb3999508938c84668d49b08030a228b7a4"
Sep 16 18:47:38.202413 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:38.202306    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 16 18:47:38.202413 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:38.202392    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 16 18:47:39.202456 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:39.202419    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 16 18:47:39.202884 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:39.202473    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 16 18:47:40.206872 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:40.206842    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:47:40.207232 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:40.207136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:47:40.207817 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:40.207799    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:47:45.460574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:45.460543    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"]
Sep 16 18:47:45.465840 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:45.465820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"
Sep 16 18:47:45.468678 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:45.468659    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:47:45.468797 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:45.468671    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:47:45.476456 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:45.476403    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"]
Sep 16 18:47:45.601213 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:45.601182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55f1f5a-2f11-4235-a96c-c95bef5ca1a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv\" (UID: \"d55f1f5a-2f11-4235-a96c-c95bef5ca1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"
Sep 16 18:47:45.702026 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:45.701994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55f1f5a-2f11-4235-a96c-c95bef5ca1a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv\" (UID: \"d55f1f5a-2f11-4235-a96c-c95bef5ca1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"
Sep 16 18:47:45.702602 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:45.702583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55f1f5a-2f11-4235-a96c-c95bef5ca1a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv\" (UID: \"d55f1f5a-2f11-4235-a96c-c95bef5ca1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"
Sep 16 18:47:45.776725 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:45.776695    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"
Sep 16 18:47:46.051252 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:46.048850    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"]
Sep 16 18:47:46.054575 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:47:46.054546    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55f1f5a_2f11_4235_a96c_c95bef5ca1a8.slice/crio-2caabc91c81c38376f03f1c5c89d2d0456574bb8ad9f4c89674de853b8192f83 WatchSource:0}: Error finding container 2caabc91c81c38376f03f1c5c89d2d0456574bb8ad9f4c89674de853b8192f83: Status 404 returned error can't find the container with id 2caabc91c81c38376f03f1c5c89d2d0456574bb8ad9f4c89674de853b8192f83
Sep 16 18:47:46.337558 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:46.337484    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:47:46.700039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:46.699952    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv" event={"ID":"d55f1f5a-2f11-4235-a96c-c95bef5ca1a8","Type":"ContainerStarted","Data":"86036c7bd617339d4e0da7b35286bce16875ae27ca6ac7429f003890c3b3ac9e"}
Sep 16 18:47:46.700039 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:46.700008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv" event={"ID":"d55f1f5a-2f11-4235-a96c-c95bef5ca1a8","Type":"ContainerStarted","Data":"2caabc91c81c38376f03f1c5c89d2d0456574bb8ad9f4c89674de853b8192f83"}
Sep 16 18:47:46.719999 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:46.719919    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv" podStartSLOduration=1.719893439 podStartE2EDuration="1.719893439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:46.718094054 +0000 UTC m=+1209.427333448" watchObservedRunningTime="2026-09-16 18:47:46.719893439 +0000 UTC m=+1209.429132846"
Sep 16 18:47:47.706812 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:47.706772    2580 generic.go:356] "Generic (PLEG): container finished" podID="d55f1f5a-2f11-4235-a96c-c95bef5ca1a8" containerID="86036c7bd617339d4e0da7b35286bce16875ae27ca6ac7429f003890c3b3ac9e" exitCode=0
Sep 16 18:47:47.707192 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:47.706856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv" event={"ID":"d55f1f5a-2f11-4235-a96c-c95bef5ca1a8","Type":"ContainerDied","Data":"86036c7bd617339d4e0da7b35286bce16875ae27ca6ac7429f003890c3b3ac9e"}
Sep 16 18:47:48.896808 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:48.896784    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"
Sep 16 18:47:48.936317 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:48.936279    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55f1f5a-2f11-4235-a96c-c95bef5ca1a8-maas-api-service-ca\") pod \"d55f1f5a-2f11-4235-a96c-c95bef5ca1a8\" (UID: \"d55f1f5a-2f11-4235-a96c-c95bef5ca1a8\") "
Sep 16 18:47:48.936766 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:48.936743    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55f1f5a-2f11-4235-a96c-c95bef5ca1a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d55f1f5a-2f11-4235-a96c-c95bef5ca1a8" (UID: "d55f1f5a-2f11-4235-a96c-c95bef5ca1a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:49.037229 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:49.037203    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55f1f5a-2f11-4235-a96c-c95bef5ca1a8-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.720584 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:49.720553    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"
Sep 16 18:47:49.720746 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:49.720580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv" event={"ID":"d55f1f5a-2f11-4235-a96c-c95bef5ca1a8","Type":"ContainerDied","Data":"2caabc91c81c38376f03f1c5c89d2d0456574bb8ad9f4c89674de853b8192f83"}
Sep 16 18:47:49.720746 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:49.720650    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2caabc91c81c38376f03f1c5c89d2d0456574bb8ad9f4c89674de853b8192f83"
Sep 16 18:47:53.756993 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:53.756910    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"]
Sep 16 18:47:53.757439 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:53.757121    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy" containerID="cri-o://12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6" gracePeriod=30
Sep 16 18:47:54.689646 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.689595    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"]
Sep 16 18:47:54.690014 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.690000    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55f1f5a-2f11-4235-a96c-c95bef5ca1a8" containerName="revoke-keys"
Sep 16 18:47:54.690014 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.690014    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55f1f5a-2f11-4235-a96c-c95bef5ca1a8" containerName="revoke-keys"
Sep 16 18:47:54.690217 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.690086    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55f1f5a-2f11-4235-a96c-c95bef5ca1a8" containerName="revoke-keys"
Sep 16 18:47:54.693430 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.693410    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.696208 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.696183    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-af5e21-openshift-default-dockercfg-zcffz\""
Sep 16 18:47:54.698467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.698442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-socket\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.698573 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.698495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccf0514-4996-492c-b238-40c982aefa41-istiod-ca-cert\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.698639 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.698575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-istio-token\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.698639 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.698601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-data\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.698639 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.698624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccf0514-4996-492c-b238-40c982aefa41-istio-podinfo\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.698793 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.698678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-certs\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.698793 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.698703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-credential-socket\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.698793 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.698769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-envoy\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.698945 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.698795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67ns5\" (UniqueName: \"kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-kube-api-access-67ns5\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.730891 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.730862    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"]
Sep 16 18:47:54.800247 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.800206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccf0514-4996-492c-b238-40c982aefa41-istiod-ca-cert\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.800742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.800283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-istio-token\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.800742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.800317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-data\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.800742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.800368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccf0514-4996-492c-b238-40c982aefa41-istio-podinfo\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.800742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.800436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-certs\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.800742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.800454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-credential-socket\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.800742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.800541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-envoy\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.800742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.800569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67ns5\" (UniqueName: \"kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-kube-api-access-67ns5\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.800742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.800606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-socket\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.800742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.800734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-data\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.801213 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.801005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccf0514-4996-492c-b238-40c982aefa41-istiod-ca-cert\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.801213 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.801081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-socket\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.801213 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.801147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-certs\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.801434 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.801412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-credential-socket\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.803072 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.803049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccf0514-4996-492c-b238-40c982aefa41-istio-podinfo\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.803199 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.803177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-envoy\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.838786 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.838754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-istio-token\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:54.838786 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:54.838773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67ns5\" (UniqueName: \"kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-kube-api-access-67ns5\") pod \"e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") " pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:55.006053 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:55.005974    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:55.298866 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:55.298841    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"]
Sep 16 18:47:55.299847 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:47:55.299826    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccf0514_4996_492c_b238_40c982aefa41.slice/crio-35be3b83ed8f59de77877028c98faf53798b18aaaba73a43afc3ce7da13c6911 WatchSource:0}: Error finding container 35be3b83ed8f59de77877028c98faf53798b18aaaba73a43afc3ce7da13c6911: Status 404 returned error can't find the container with id 35be3b83ed8f59de77877028c98faf53798b18aaaba73a43afc3ce7da13c6911
Sep 16 18:47:55.302304 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:55.302271    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:55.302429 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:55.302366    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:55.302429 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:55.302412    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:55.760885 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:55.760777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s" event={"ID":"4ccf0514-4996-492c-b238-40c982aefa41","Type":"ContainerStarted","Data":"3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb"}
Sep 16 18:47:55.760885 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:55.760872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s" event={"ID":"4ccf0514-4996-492c-b238-40c982aefa41","Type":"ContainerStarted","Data":"35be3b83ed8f59de77877028c98faf53798b18aaaba73a43afc3ce7da13c6911"}
Sep 16 18:47:55.798158 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:55.798108    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s" podStartSLOduration=1.798092187 podStartE2EDuration="1.798092187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:55.795668716 +0000 UTC m=+1218.504908127" watchObservedRunningTime="2026-09-16 18:47:55.798092187 +0000 UTC m=+1218.507331579"
Sep 16 18:47:56.006363 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.006317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:56.012821 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.012745    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:56.575913 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.575888    2580 patch_prober.go:28] interesting pod/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 18:47:56.576045 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.575938    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 18:47:56.768697 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.768667    2580 generic.go:356] "Generic (PLEG): container finished" podID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerID="82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8" exitCode=0
Sep 16 18:47:56.768854 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.768736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" event={"ID":"7eb74c06-dff3-4c09-a7e0-cf2a86e74937","Type":"ContainerDied","Data":"82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8"}
Sep 16 18:47:56.769180 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.769159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:56.769407 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.769307    2580 scope.go:117] "RemoveContainer" containerID="82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8"
Sep 16 18:47:56.770386 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.770369    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:47:56.770750 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.770723    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:56.770807 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.770794    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:56.770843 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:56.770830    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 18:47:57.802267 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:57.802217    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" event={"ID":"7eb74c06-dff3-4c09-a7e0-cf2a86e74937","Type":"ContainerStarted","Data":"148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a"}
Sep 16 18:47:58.075416 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:58.074386    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"]
Sep 16 18:47:58.075416 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:58.074735    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6f41de-hpjcv"]
Sep 16 18:47:58.472582 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:58.472484    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:47:58.472941 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:58.472908    2580 patch_prober.go:28] interesting pod/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 18:47:58.473063 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:58.473016    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 18:47:59.046265 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.046242    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:47:59.149395 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149362    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-socket\") pod \"93b77c1a-cc53-4a16-b737-abbe7eec3103\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") "
Sep 16 18:47:59.149566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrbsj\" (UniqueName: \"kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-kube-api-access-mrbsj\") pod \"93b77c1a-cc53-4a16-b737-abbe7eec3103\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") "
Sep 16 18:47:59.149566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-certs\") pod \"93b77c1a-cc53-4a16-b737-abbe7eec3103\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") "
Sep 16 18:47:59.149566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-podinfo\") pod \"93b77c1a-cc53-4a16-b737-abbe7eec3103\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") "
Sep 16 18:47:59.149566 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149543    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-envoy\") pod \"93b77c1a-cc53-4a16-b737-abbe7eec3103\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") "
Sep 16 18:47:59.149786 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149568    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-token\") pod \"93b77c1a-cc53-4a16-b737-abbe7eec3103\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") "
Sep 16 18:47:59.149786 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149662    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "93b77c1a-cc53-4a16-b737-abbe7eec3103" (UID: "93b77c1a-cc53-4a16-b737-abbe7eec3103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:59.149786 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149690    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-data\") pod \"93b77c1a-cc53-4a16-b737-abbe7eec3103\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") "
Sep 16 18:47:59.149786 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b77c1a-cc53-4a16-b737-abbe7eec3103-istiod-ca-cert\") pod \"93b77c1a-cc53-4a16-b737-abbe7eec3103\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") "
Sep 16 18:47:59.149786 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149757    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "93b77c1a-cc53-4a16-b737-abbe7eec3103" (UID: "93b77c1a-cc53-4a16-b737-abbe7eec3103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:59.149786 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149774    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-credential-socket\") pod \"93b77c1a-cc53-4a16-b737-abbe7eec3103\" (UID: \"93b77c1a-cc53-4a16-b737-abbe7eec3103\") "
Sep 16 18:47:59.150092 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.149969    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-data" (OuterVolumeSpecName: "istio-data") pod "93b77c1a-cc53-4a16-b737-abbe7eec3103" (UID: "93b77c1a-cc53-4a16-b737-abbe7eec3103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:59.150168 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.150070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b77c1a-cc53-4a16-b737-abbe7eec3103-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "93b77c1a-cc53-4a16-b737-abbe7eec3103" (UID: "93b77c1a-cc53-4a16-b737-abbe7eec3103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:59.150168 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.150143    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "93b77c1a-cc53-4a16-b737-abbe7eec3103" (UID: "93b77c1a-cc53-4a16-b737-abbe7eec3103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:59.150274 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.150204    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.150274 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.150226    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-data\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.150274 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.150243    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-workload-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.151840 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.151811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "93b77c1a-cc53-4a16-b737-abbe7eec3103" (UID: "93b77c1a-cc53-4a16-b737-abbe7eec3103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:47:59.151991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.151974    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "93b77c1a-cc53-4a16-b737-abbe7eec3103" (UID: "93b77c1a-cc53-4a16-b737-abbe7eec3103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:59.152056 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.152005    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-kube-api-access-mrbsj" (OuterVolumeSpecName: "kube-api-access-mrbsj") pod "93b77c1a-cc53-4a16-b737-abbe7eec3103" (UID: "93b77c1a-cc53-4a16-b737-abbe7eec3103"). InnerVolumeSpecName "kube-api-access-mrbsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:59.152119 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.152073    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-token" (OuterVolumeSpecName: "istio-token") pod "93b77c1a-cc53-4a16-b737-abbe7eec3103" (UID: "93b77c1a-cc53-4a16-b737-abbe7eec3103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:59.251455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.251421    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-podinfo\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.251455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.251451    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-envoy\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.251455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.251460    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-istio-token\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.251672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.251466    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b77c1a-cc53-4a16-b737-abbe7eec3103-istiod-ca-cert\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.251672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.251475    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b77c1a-cc53-4a16-b737-abbe7eec3103-credential-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.251672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.251485    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrbsj\" (UniqueName: \"kubernetes.io/projected/93b77c1a-cc53-4a16-b737-abbe7eec3103-kube-api-access-mrbsj\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.473022 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.472922    2580 patch_prober.go:28] interesting pod/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 18:47:59.473022 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.472993    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 18:47:59.788088 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.788038    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d55f1f5a-2f11-4235-a96c-c95bef5ca1a8" path="/var/lib/kubelet/pods/d55f1f5a-2f11-4235-a96c-c95bef5ca1a8/volumes"
Sep 16 18:47:59.815469 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.815439    2580 generic.go:356] "Generic (PLEG): container finished" podID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerID="12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6" exitCode=0
Sep 16 18:47:59.815577 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.815490    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"
Sep 16 18:47:59.815577 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.815521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" event={"ID":"93b77c1a-cc53-4a16-b737-abbe7eec3103","Type":"ContainerDied","Data":"12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6"}
Sep 16 18:47:59.815577 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.815570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65" event={"ID":"93b77c1a-cc53-4a16-b737-abbe7eec3103","Type":"ContainerDied","Data":"37023d3424893d640c2947e547b87dbf628b490f232c282e0a3380cffcaf786c"}
Sep 16 18:47:59.815678 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.815598    2580 scope.go:117] "RemoveContainer" containerID="12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6"
Sep 16 18:47:59.824744 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.824726    2580 scope.go:117] "RemoveContainer" containerID="412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb"
Sep 16 18:47:59.832896 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.832873    2580 scope.go:117] "RemoveContainer" containerID="12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6"
Sep 16 18:47:59.833158 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:59.833139    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6\": container with ID starting with 12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6 not found: ID does not exist" containerID="12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6"
Sep 16 18:47:59.833216 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.833167    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6"} err="failed to get container status \"12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6\": rpc error: code = NotFound desc = could not find container \"12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6\": container with ID starting with 12e7a12230fbd5c02bccbf83b91dcde3d464afeff038b1e80984796207f410b6 not found: ID does not exist"
Sep 16 18:47:59.833216 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.833184    2580 scope.go:117] "RemoveContainer" containerID="412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb"
Sep 16 18:47:59.833463 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:47:59.833447    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb\": container with ID starting with 412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb not found: ID does not exist" containerID="412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb"
Sep 16 18:47:59.833514 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.833469    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb"} err="failed to get container status \"412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb\": rpc error: code = NotFound desc = could not find container \"412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb\": container with ID starting with 412b5aa2cc485fd0650b4157ade4a53fd62d882469004aac5664a3dc07c396eb not found: ID does not exist"
Sep 16 18:47:59.836817 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.836798    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"]
Sep 16 18:47:59.845034 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:47:59.845014    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-16ae98-openshift-default-64db49fbcb-fpv65"]
Sep 16 18:48:00.472824 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:00.472368    2580 patch_prober.go:28] interesting pod/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 18:48:00.472824 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:00.472432    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 18:48:01.473142 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:01.473097    2580 patch_prober.go:28] interesting pod/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 18:48:01.473673 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:01.473173    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 18:48:01.790274 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:01.790232    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" path="/var/lib/kubelet/pods/93b77c1a-cc53-4a16-b737-abbe7eec3103/volumes"
Sep 16 18:48:02.472599 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:02.472552    2580 patch_prober.go:28] interesting pod/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 16 18:48:02.472812 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:02.472644    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 18:48:03.476894 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:03.476862    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:48:03.477272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:03.477157    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:48:03.477916 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:03.477895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:48:06.497322 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.497288    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"]
Sep 16 18:48:06.497769 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.497744    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy"
Sep 16 18:48:06.497769 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.497758    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy"
Sep 16 18:48:06.497854 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.497843    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy"
Sep 16 18:48:06.497891 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.497856    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy"
Sep 16 18:48:06.500646 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.500629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"
Sep 16 18:48:06.503218 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.503193    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:48:06.503324 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.503195    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:48:06.510731 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.510709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"]
Sep 16 18:48:06.631269 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.631231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl\" (UID: \"0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"
Sep 16 18:48:06.732130 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.732097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl\" (UID: \"0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"
Sep 16 18:48:06.732768 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.732744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl\" (UID: \"0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"
Sep 16 18:48:06.812009 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:06.811985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"
Sep 16 18:48:07.085871 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:07.085842    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"]
Sep 16 18:48:07.087456 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:48:07.087432    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eaebb77_0306_4dbe_8f50_52ee9a7cf4bb.slice/crio-368d3b18e9bfdb6157aa7cfda8f112947fe51efcb67e2ec787fd2f105cdc65da WatchSource:0}: Error finding container 368d3b18e9bfdb6157aa7cfda8f112947fe51efcb67e2ec787fd2f105cdc65da: Status 404 returned error can't find the container with id 368d3b18e9bfdb6157aa7cfda8f112947fe51efcb67e2ec787fd2f105cdc65da
Sep 16 18:48:07.872527 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:07.872492    2580 generic.go:356] "Generic (PLEG): container finished" podID="0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb" containerID="c04c81a0a1e2b04d8ec70e73000e29628db7e0be7a8c774ef35ea81511459a54" exitCode=0
Sep 16 18:48:07.872952 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:07.872545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl" event={"ID":"0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb","Type":"ContainerDied","Data":"c04c81a0a1e2b04d8ec70e73000e29628db7e0be7a8c774ef35ea81511459a54"}
Sep 16 18:48:07.872952 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:07.872598    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl" event={"ID":"0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb","Type":"ContainerStarted","Data":"368d3b18e9bfdb6157aa7cfda8f112947fe51efcb67e2ec787fd2f105cdc65da"}
Sep 16 18:48:09.012503 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:09.012480    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"
Sep 16 18:48:09.156993 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:09.156909    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb-maas-api-service-ca\") pod \"0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb\" (UID: \"0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb\") "
Sep 16 18:48:09.157268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:09.157246    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb" (UID: "0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:09.258377 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:09.258325    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:48:09.886155 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:09.886122    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"
Sep 16 18:48:09.886313 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:09.886154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl" event={"ID":"0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb","Type":"ContainerDied","Data":"368d3b18e9bfdb6157aa7cfda8f112947fe51efcb67e2ec787fd2f105cdc65da"}
Sep 16 18:48:09.886313 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:09.886203    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="368d3b18e9bfdb6157aa7cfda8f112947fe51efcb67e2ec787fd2f105cdc65da"
Sep 16 18:48:18.633275 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:18.633241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:48:19.690808 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:19.690770    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"]
Sep 16 18:48:20.580258 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:20.580227    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp"]
Sep 16 18:48:20.583789 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:20.583763    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e8510f-29826405-jpmnp"]
Sep 16 18:48:20.936006 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:20.935920    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"]
Sep 16 18:48:20.941585 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:20.941547    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8510f-8xmbl"]
Sep 16 18:48:21.783258 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:21.783218    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb" path="/var/lib/kubelet/pods/0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb/volumes"
Sep 16 18:48:21.783577 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:21.783562    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c0df27b-d20a-4d5b-843a-327145560f55" path="/var/lib/kubelet/pods/5c0df27b-d20a-4d5b-843a-327145560f55/volumes"
Sep 16 18:48:34.710214 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:34.710161    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq" podUID="2fe9a490-1875-483c-bc61-0c2af3c83b34" containerName="main" containerID="cri-o://845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b" gracePeriod=45
Sep 16 18:48:34.969580 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:34.969525    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:48:35.001604 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.001578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-dshm\") pod \"2fe9a490-1875-483c-bc61-0c2af3c83b34\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") "
Sep 16 18:48:35.001789 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.001616    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe9a490-1875-483c-bc61-0c2af3c83b34-tls-certs\") pod \"2fe9a490-1875-483c-bc61-0c2af3c83b34\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") "
Sep 16 18:48:35.001789 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.001771    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-home\") pod \"2fe9a490-1875-483c-bc61-0c2af3c83b34\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") "
Sep 16 18:48:35.001922 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.001828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-tmp-dir\") pod \"2fe9a490-1875-483c-bc61-0c2af3c83b34\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") "
Sep 16 18:48:35.001922 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.001898    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-model-cache\") pod \"2fe9a490-1875-483c-bc61-0c2af3c83b34\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") "
Sep 16 18:48:35.002033 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.001937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv27w\" (UniqueName: \"kubernetes.io/projected/2fe9a490-1875-483c-bc61-0c2af3c83b34-kube-api-access-fv27w\") pod \"2fe9a490-1875-483c-bc61-0c2af3c83b34\" (UID: \"2fe9a490-1875-483c-bc61-0c2af3c83b34\") "
Sep 16 18:48:35.002099 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.002043    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-home" (OuterVolumeSpecName: "home") pod "2fe9a490-1875-483c-bc61-0c2af3c83b34" (UID: "2fe9a490-1875-483c-bc61-0c2af3c83b34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:35.002157 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.002111    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fe9a490-1875-483c-bc61-0c2af3c83b34" (UID: "2fe9a490-1875-483c-bc61-0c2af3c83b34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:35.002214 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.002183    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fe9a490-1875-483c-bc61-0c2af3c83b34" (UID: "2fe9a490-1875-483c-bc61-0c2af3c83b34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:35.002429 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.002405    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-home\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.002546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.002455    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-tmp-dir\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.002546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.002469    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-model-cache\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.004018 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.003985    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-dshm" (OuterVolumeSpecName: "dshm") pod "2fe9a490-1875-483c-bc61-0c2af3c83b34" (UID: "2fe9a490-1875-483c-bc61-0c2af3c83b34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:35.004164 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.004146    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe9a490-1875-483c-bc61-0c2af3c83b34-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fe9a490-1875-483c-bc61-0c2af3c83b34" (UID: "2fe9a490-1875-483c-bc61-0c2af3c83b34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:35.004214 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.004178    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe9a490-1875-483c-bc61-0c2af3c83b34-kube-api-access-fv27w" (OuterVolumeSpecName: "kube-api-access-fv27w") pod "2fe9a490-1875-483c-bc61-0c2af3c83b34" (UID: "2fe9a490-1875-483c-bc61-0c2af3c83b34"). InnerVolumeSpecName "kube-api-access-fv27w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:35.031219 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.031192    2580 generic.go:356] "Generic (PLEG): container finished" podID="2fe9a490-1875-483c-bc61-0c2af3c83b34" containerID="845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b" exitCode=0
Sep 16 18:48:35.031382 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.031238    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"
Sep 16 18:48:35.031382 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.031277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq" event={"ID":"2fe9a490-1875-483c-bc61-0c2af3c83b34","Type":"ContainerDied","Data":"845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b"}
Sep 16 18:48:35.031382 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.031326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq" event={"ID":"2fe9a490-1875-483c-bc61-0c2af3c83b34","Type":"ContainerDied","Data":"157f83a051fde53f8f47813e57ac5f16b3bc6026d5415780c3420a2c7bd6758b"}
Sep 16 18:48:35.031382 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.031351    2580 scope.go:117] "RemoveContainer" containerID="845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b"
Sep 16 18:48:35.040661 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.040646    2580 scope.go:117] "RemoveContainer" containerID="845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b"
Sep 16 18:48:35.040884 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:48:35.040863    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b\": container with ID starting with 845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b not found: ID does not exist" containerID="845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b"
Sep 16 18:48:35.040946 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.040895    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b"} err="failed to get container status \"845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b\": rpc error: code = NotFound desc = could not find container \"845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b\": container with ID starting with 845c44daf97efac5c054b2d41a5f3dd94ebc9fded76c55cd5ef39c594f7ecc7b not found: ID does not exist"
Sep 16 18:48:35.061559 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.061523    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"]
Sep 16 18:48:35.071524 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.071495    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e8510f/test-model-e8510f-kserve-56757f7bcf-dfvwq"]
Sep 16 18:48:35.102957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.102928    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv27w\" (UniqueName: \"kubernetes.io/projected/2fe9a490-1875-483c-bc61-0c2af3c83b34-kube-api-access-fv27w\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.102957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.102953    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe9a490-1875-483c-bc61-0c2af3c83b34-dshm\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.103109 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.102963    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe9a490-1875-483c-bc61-0c2af3c83b34-tls-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.783126 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:35.783091    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fe9a490-1875-483c-bc61-0c2af3c83b34" path="/var/lib/kubelet/pods/2fe9a490-1875-483c-bc61-0c2af3c83b34/volumes"
Sep 16 18:48:37.732671 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:37.732638    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:48:37.933382 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:48:37.933354    2580 scope.go:117] "RemoveContainer" containerID="cbc12192dee367a94d68282ab9e47dcd3939cc2487de4438ed99e02a4189e306"
Sep 16 18:49:05.533588 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:05.533556    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:49:13.027751 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.027716    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"]
Sep 16 18:49:13.028224 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.028131    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb" containerName="revoke-keys"
Sep 16 18:49:13.028224 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.028143    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb" containerName="revoke-keys"
Sep 16 18:49:13.028224 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.028155    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy"
Sep 16 18:49:13.028224 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.028161    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b77c1a-cc53-4a16-b737-abbe7eec3103" containerName="istio-proxy"
Sep 16 18:49:13.028224 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.028170    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fe9a490-1875-483c-bc61-0c2af3c83b34" containerName="main"
Sep 16 18:49:13.028224 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.028175    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fe9a490-1875-483c-bc61-0c2af3c83b34" containerName="main"
Sep 16 18:49:13.028450 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.028235    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eaebb77-0306-4dbe-8f50-52ee9a7cf4bb" containerName="revoke-keys"
Sep 16 18:49:13.028450 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.028242    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fe9a490-1875-483c-bc61-0c2af3c83b34" containerName="main"
Sep 16 18:49:13.030413 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.030397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.033899 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.033875    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-af5e21\"/\"noten-model-af5e21-kserve-self-signed-certs\""
Sep 16 18:49:13.034031 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.033903    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-af5e21\"/\"kube-root-ca.crt\""
Sep 16 18:49:13.034031 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.033903    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-af5e21\"/\"openshift-service-ca.crt\""
Sep 16 18:49:13.034031 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.033903    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-af5e21\"/\"default-dockercfg-dzmmh\""
Sep 16 18:49:13.042975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.042953    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"]
Sep 16 18:49:13.072421 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.072397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-tmp-dir\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.072548 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.072438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-dshm\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.072548 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.072464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-home\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.072548 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.072519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a515228d-d41c-48e0-932c-397d72b4be57-tls-certs\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.072656 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.072620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-model-cache\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.072656 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.072652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9sk\" (UniqueName: \"kubernetes.io/projected/a515228d-d41c-48e0-932c-397d72b4be57-kube-api-access-hx9sk\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.173097 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.173059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-model-cache\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.173248 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.173110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9sk\" (UniqueName: \"kubernetes.io/projected/a515228d-d41c-48e0-932c-397d72b4be57-kube-api-access-hx9sk\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.173248 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.173145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-tmp-dir\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.173248 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.173176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-dshm\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.173479 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.173313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-home\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.173479 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.173374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a515228d-d41c-48e0-932c-397d72b4be57-tls-certs\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.173479 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.173459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-model-cache\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.173612 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.173501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-tmp-dir\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.173612 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.173582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-home\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.175314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.175292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-dshm\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.175745 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.175728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a515228d-d41c-48e0-932c-397d72b4be57-tls-certs\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.182416 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.182397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9sk\" (UniqueName: \"kubernetes.io/projected/a515228d-d41c-48e0-932c-397d72b4be57-kube-api-access-hx9sk\") pod \"noten-model-af5e21-kserve-59cf955c9f-5glm9\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") " pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.339249 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.339173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:13.612968 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:13.612941    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"]
Sep 16 18:49:13.614604 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:49:13.614577    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda515228d_d41c_48e0_932c_397d72b4be57.slice/crio-a75dfb05042d7bacd6193866f9b6fee7b6f6c3bdd38830fe97f089a0b3724d8b WatchSource:0}: Error finding container a75dfb05042d7bacd6193866f9b6fee7b6f6c3bdd38830fe97f089a0b3724d8b: Status 404 returned error can't find the container with id a75dfb05042d7bacd6193866f9b6fee7b6f6c3bdd38830fe97f089a0b3724d8b
Sep 16 18:49:14.256393 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:14.256357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9" event={"ID":"a515228d-d41c-48e0-932c-397d72b4be57","Type":"ContainerStarted","Data":"57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d"}
Sep 16 18:49:14.256773 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:14.256405    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9" event={"ID":"a515228d-d41c-48e0-932c-397d72b4be57","Type":"ContainerStarted","Data":"a75dfb05042d7bacd6193866f9b6fee7b6f6c3bdd38830fe97f089a0b3724d8b"}
Sep 16 18:49:14.274587 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:14.274539    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9" podStartSLOduration=1.274523967 podStartE2EDuration="1.274523967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:14.273910794 +0000 UTC m=+1296.983150204" watchObservedRunningTime="2026-09-16 18:49:14.274523967 +0000 UTC m=+1296.983763420"
Sep 16 18:49:23.339970 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:23.339936    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:23.340498 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:23.339990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:23.352934 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:23.352912    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:24.325070 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:24.325039    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:49:25.931187 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:25.931152    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"]
Sep 16 18:49:25.934487 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:25.934470    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:25.939280 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:25.939254    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4dda98\"/\"distinct-2-w0-4dda98-kserve-self-signed-certs\""
Sep 16 18:49:25.949894 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:25.949874    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"]
Sep 16 18:49:26.001538 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.001510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmxbv\" (UniqueName: \"kubernetes.io/projected/4b254da9-78f9-457a-bf98-97a9cadfec99-kube-api-access-hmxbv\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.001691 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.001568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-model-cache\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.001691 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.001597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-dshm\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.001816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.001690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-home\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.001816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.001738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-tmp-dir\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.001816 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.001800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b254da9-78f9-457a-bf98-97a9cadfec99-tls-certs\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.102720 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.102687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxbv\" (UniqueName: \"kubernetes.io/projected/4b254da9-78f9-457a-bf98-97a9cadfec99-kube-api-access-hmxbv\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.102919 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.102760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-model-cache\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.102919 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.102782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-dshm\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.102919 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.102824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-home\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.102919 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.102861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-tmp-dir\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.102919 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.102903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b254da9-78f9-457a-bf98-97a9cadfec99-tls-certs\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.103200 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.103170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-model-cache\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.103318 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.103229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-home\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.103318 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.103277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-tmp-dir\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.105018 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.104989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-dshm\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.105262 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.105246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b254da9-78f9-457a-bf98-97a9cadfec99-tls-certs\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.112546 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.112523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxbv\" (UniqueName: \"kubernetes.io/projected/4b254da9-78f9-457a-bf98-97a9cadfec99-kube-api-access-hmxbv\") pod \"distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") " pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.244455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.244384    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:26.522498 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:26.522472    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"]
Sep 16 18:49:26.524314 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:49:26.524276    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b254da9_78f9_457a_bf98_97a9cadfec99.slice/crio-6d014d6a7894e46d62e219db8e5aa7995ae94fcae8e418259090787bc864a7e9 WatchSource:0}: Error finding container 6d014d6a7894e46d62e219db8e5aa7995ae94fcae8e418259090787bc864a7e9: Status 404 returned error can't find the container with id 6d014d6a7894e46d62e219db8e5aa7995ae94fcae8e418259090787bc864a7e9
Sep 16 18:49:27.332815 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:27.332783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c" event={"ID":"4b254da9-78f9-457a-bf98-97a9cadfec99","Type":"ContainerStarted","Data":"9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736"}
Sep 16 18:49:27.333221 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:27.332824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c" event={"ID":"4b254da9-78f9-457a-bf98-97a9cadfec99","Type":"ContainerStarted","Data":"6d014d6a7894e46d62e219db8e5aa7995ae94fcae8e418259090787bc864a7e9"}
Sep 16 18:49:27.355261 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:27.355212    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c" podStartSLOduration=2.355198297 podStartE2EDuration="2.355198297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:27.353402639 +0000 UTC m=+1310.062642027" watchObservedRunningTime="2026-09-16 18:49:27.355198297 +0000 UTC m=+1310.064437688"
Sep 16 18:49:29.265154 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:29.265117    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:49:36.244645 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:36.244590    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:36.244645 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:36.244661    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:36.256958 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:36.256931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:36.396296 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:36.396269    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:49:37.962848 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:37.962815    2580 scope.go:117] "RemoveContainer" containerID="b18e9777149d02d392f8bed281fd88bc9117327e762437267159360a7c1324f3"
Sep 16 18:49:37.971660 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:37.971636    2580 scope.go:117] "RemoveContainer" containerID="2dd23e00e35106169010fed7c03c06ea296b5c1c4640e2fa4417ba8cc9369895"
Sep 16 18:49:37.980124 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:37.980105    2580 scope.go:117] "RemoveContainer" containerID="da62a2258b64052cd672fa2a7644e50d50c2dd9d7d6dae3e1a47ab148173988f"
Sep 16 18:49:39.155609 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.155572    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"]
Sep 16 18:49:39.160567 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.160542    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 18:49:39.162952 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.162927    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"]
Sep 16 18:49:39.163063 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.163006    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e65cc4\""
Sep 16 18:49:39.166820 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.166800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 18:49:39.171247 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.171218    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e65cc4-dockercfg-htpdh\""
Sep 16 18:49:39.174496 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.174468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"]
Sep 16 18:49:39.177503 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.177479    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"]
Sep 16 18:49:39.237715 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.237679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6375c0-59f8-4f29-864f-ceeaf950042f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd\" (UID: \"8a6375c0-59f8-4f29-864f-ceeaf950042f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 18:49:39.237715 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.237716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vvj\" (UniqueName: \"kubernetes.io/projected/bce64fa3-476d-42a0-8b1f-997cdb57da4e-kube-api-access-69vvj\") pod \"payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk\" (UID: \"bce64fa3-476d-42a0-8b1f-997cdb57da4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 18:49:39.237952 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.237739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7sdv\" (UniqueName: \"kubernetes.io/projected/8a6375c0-59f8-4f29-864f-ceeaf950042f-kube-api-access-f7sdv\") pod \"payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd\" (UID: \"8a6375c0-59f8-4f29-864f-ceeaf950042f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 18:49:39.237952 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.237851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce64fa3-476d-42a0-8b1f-997cdb57da4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk\" (UID: \"bce64fa3-476d-42a0-8b1f-997cdb57da4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 18:49:39.339311 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.339279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6375c0-59f8-4f29-864f-ceeaf950042f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd\" (UID: \"8a6375c0-59f8-4f29-864f-ceeaf950042f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 18:49:39.339529 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.339321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69vvj\" (UniqueName: \"kubernetes.io/projected/bce64fa3-476d-42a0-8b1f-997cdb57da4e-kube-api-access-69vvj\") pod \"payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk\" (UID: \"bce64fa3-476d-42a0-8b1f-997cdb57da4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 18:49:39.339529 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.339380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7sdv\" (UniqueName: \"kubernetes.io/projected/8a6375c0-59f8-4f29-864f-ceeaf950042f-kube-api-access-f7sdv\") pod \"payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd\" (UID: \"8a6375c0-59f8-4f29-864f-ceeaf950042f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 18:49:39.339529 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.339451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce64fa3-476d-42a0-8b1f-997cdb57da4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk\" (UID: \"bce64fa3-476d-42a0-8b1f-997cdb57da4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 18:49:39.340051 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.340024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6375c0-59f8-4f29-864f-ceeaf950042f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd\" (UID: \"8a6375c0-59f8-4f29-864f-ceeaf950042f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 18:49:39.340205 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.340185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce64fa3-476d-42a0-8b1f-997cdb57da4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk\" (UID: \"bce64fa3-476d-42a0-8b1f-997cdb57da4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 18:49:39.353067 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.353041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vvj\" (UniqueName: \"kubernetes.io/projected/bce64fa3-476d-42a0-8b1f-997cdb57da4e-kube-api-access-69vvj\") pod \"payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk\" (UID: \"bce64fa3-476d-42a0-8b1f-997cdb57da4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 18:49:39.353679 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.353656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7sdv\" (UniqueName: \"kubernetes.io/projected/8a6375c0-59f8-4f29-864f-ceeaf950042f-kube-api-access-f7sdv\") pod \"payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd\" (UID: \"8a6375c0-59f8-4f29-864f-ceeaf950042f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 18:49:39.472524 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.472430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 18:49:39.482549 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.482516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 18:49:39.748525 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.748493    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"]
Sep 16 18:49:39.750883 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:49:39.750857    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6375c0_59f8_4f29_864f_ceeaf950042f.slice/crio-3d5c71bf250052d78e6984f593ab141ffc4df78c91a46a8c401601b5d4f2be4c WatchSource:0}: Error finding container 3d5c71bf250052d78e6984f593ab141ffc4df78c91a46a8c401601b5d4f2be4c: Status 404 returned error can't find the container with id 3d5c71bf250052d78e6984f593ab141ffc4df78c91a46a8c401601b5d4f2be4c
Sep 16 18:49:39.787643 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:49:39.787618    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce64fa3_476d_42a0_8b1f_997cdb57da4e.slice/crio-ec5f1f97b6ae569054e3dfbb57f0bb2f5504eed8e8d052ed85474ea8da787352 WatchSource:0}: Error finding container ec5f1f97b6ae569054e3dfbb57f0bb2f5504eed8e8d052ed85474ea8da787352: Status 404 returned error can't find the container with id ec5f1f97b6ae569054e3dfbb57f0bb2f5504eed8e8d052ed85474ea8da787352
Sep 16 18:49:39.789036 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:39.789015    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"]
Sep 16 18:49:40.409530 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:40.409488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd" event={"ID":"8a6375c0-59f8-4f29-864f-ceeaf950042f","Type":"ContainerStarted","Data":"18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1"}
Sep 16 18:49:40.409956 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:40.409681    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 18:49:40.409956 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:40.409727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd" event={"ID":"8a6375c0-59f8-4f29-864f-ceeaf950042f","Type":"ContainerStarted","Data":"3d5c71bf250052d78e6984f593ab141ffc4df78c91a46a8c401601b5d4f2be4c"}
Sep 16 18:49:40.411202 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:40.411176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk" event={"ID":"bce64fa3-476d-42a0-8b1f-997cdb57da4e","Type":"ContainerStarted","Data":"cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0"}
Sep 16 18:49:40.411305 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:40.411214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk" event={"ID":"bce64fa3-476d-42a0-8b1f-997cdb57da4e","Type":"ContainerStarted","Data":"ec5f1f97b6ae569054e3dfbb57f0bb2f5504eed8e8d052ed85474ea8da787352"}
Sep 16 18:49:40.411305 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:40.411236    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 18:49:40.430168 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:40.430120    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd" podStartSLOduration=1.430104292 podStartE2EDuration="1.430104292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:40.429638977 +0000 UTC m=+1323.138878370" watchObservedRunningTime="2026-09-16 18:49:40.430104292 +0000 UTC m=+1323.139343685"
Sep 16 18:49:40.451251 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:40.451209    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk" podStartSLOduration=1.4511956480000001 podStartE2EDuration="1.451195648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:40.449627368 +0000 UTC m=+1323.158866761" watchObservedRunningTime="2026-09-16 18:49:40.451195648 +0000 UTC m=+1323.160435040"
Sep 16 18:49:44.623446 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:44.623415    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"]
Sep 16 18:49:44.623967 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:44.623637    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s" podUID="4ccf0514-4996-492c-b238-40c982aefa41" containerName="istio-proxy" containerID="cri-o://3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb" gracePeriod=30
Sep 16 18:49:49.957403 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:49.957373    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:49:50.015646 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.015608    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"]
Sep 16 18:49:50.046511 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.046420    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-istio-token\") pod \"4ccf0514-4996-492c-b238-40c982aefa41\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") "
Sep 16 18:49:50.046511 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.046498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-socket\") pod \"4ccf0514-4996-492c-b238-40c982aefa41\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") "
Sep 16 18:49:50.046771 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.046547    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-data\") pod \"4ccf0514-4996-492c-b238-40c982aefa41\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") "
Sep 16 18:49:50.046771 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.046591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccf0514-4996-492c-b238-40c982aefa41-istio-podinfo\") pod \"4ccf0514-4996-492c-b238-40c982aefa41\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") "
Sep 16 18:49:50.046771 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.046634    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-envoy\") pod \"4ccf0514-4996-492c-b238-40c982aefa41\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") "
Sep 16 18:49:50.046771 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.046665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccf0514-4996-492c-b238-40c982aefa41-istiod-ca-cert\") pod \"4ccf0514-4996-492c-b238-40c982aefa41\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") "
Sep 16 18:49:50.046771 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.046717    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67ns5\" (UniqueName: \"kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-kube-api-access-67ns5\") pod \"4ccf0514-4996-492c-b238-40c982aefa41\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") "
Sep 16 18:49:50.046771 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.046752    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-credential-socket\") pod \"4ccf0514-4996-492c-b238-40c982aefa41\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") "
Sep 16 18:49:50.047099 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.046794    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-certs\") pod \"4ccf0514-4996-492c-b238-40c982aefa41\" (UID: \"4ccf0514-4996-492c-b238-40c982aefa41\") "
Sep 16 18:49:50.047662 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.047400    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ccf0514-4996-492c-b238-40c982aefa41" (UID: "4ccf0514-4996-492c-b238-40c982aefa41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:50.047662 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.047469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ccf0514-4996-492c-b238-40c982aefa41" (UID: "4ccf0514-4996-492c-b238-40c982aefa41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:50.047662 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.047555    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ccf0514-4996-492c-b238-40c982aefa41" (UID: "4ccf0514-4996-492c-b238-40c982aefa41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:50.047662 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.047628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccf0514-4996-492c-b238-40c982aefa41-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ccf0514-4996-492c-b238-40c982aefa41" (UID: "4ccf0514-4996-492c-b238-40c982aefa41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:50.047662 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.047656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ccf0514-4996-492c-b238-40c982aefa41" (UID: "4ccf0514-4996-492c-b238-40c982aefa41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:50.049451 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.049425    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ccf0514-4996-492c-b238-40c982aefa41-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ccf0514-4996-492c-b238-40c982aefa41" (UID: "4ccf0514-4996-492c-b238-40c982aefa41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:49:50.049780 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.049751    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-kube-api-access-67ns5" (OuterVolumeSpecName: "kube-api-access-67ns5") pod "4ccf0514-4996-492c-b238-40c982aefa41" (UID: "4ccf0514-4996-492c-b238-40c982aefa41"). InnerVolumeSpecName "kube-api-access-67ns5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:50.049888 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.049758    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ccf0514-4996-492c-b238-40c982aefa41" (UID: "4ccf0514-4996-492c-b238-40c982aefa41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:50.050098 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.050074    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ccf0514-4996-492c-b238-40c982aefa41" (UID: "4ccf0514-4996-492c-b238-40c982aefa41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:50.148434 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.148393    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67ns5\" (UniqueName: \"kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-kube-api-access-67ns5\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:49:50.148434 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.148426    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-credential-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:49:50.148434 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.148436    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:49:50.148434 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.148445    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccf0514-4996-492c-b238-40c982aefa41-istio-token\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:49:50.148700 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.148454    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-workload-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:49:50.148700 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.148462    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-data\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:49:50.148700 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.148469    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccf0514-4996-492c-b238-40c982aefa41-istio-podinfo\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:49:50.148700 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.148477    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccf0514-4996-492c-b238-40c982aefa41-istio-envoy\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:49:50.148700 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.148483    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccf0514-4996-492c-b238-40c982aefa41-istiod-ca-cert\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:49:50.469471 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.469435    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ccf0514-4996-492c-b238-40c982aefa41" containerID="3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb" exitCode=0
Sep 16 18:49:50.469680 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.469504    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"
Sep 16 18:49:50.469680 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.469534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s" event={"ID":"4ccf0514-4996-492c-b238-40c982aefa41","Type":"ContainerDied","Data":"3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb"}
Sep 16 18:49:50.469680 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.469603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s" event={"ID":"4ccf0514-4996-492c-b238-40c982aefa41","Type":"ContainerDied","Data":"35be3b83ed8f59de77877028c98faf53798b18aaaba73a43afc3ce7da13c6911"}
Sep 16 18:49:50.469680 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.469633    2580 scope.go:117] "RemoveContainer" containerID="3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb"
Sep 16 18:49:50.490867 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.490838    2580 scope.go:117] "RemoveContainer" containerID="3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb"
Sep 16 18:49:50.491176 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:49:50.491148    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb\": container with ID starting with 3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb not found: ID does not exist" containerID="3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb"
Sep 16 18:49:50.491391 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.491179    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb"} err="failed to get container status \"3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb\": rpc error: code = NotFound desc = could not find container \"3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb\": container with ID starting with 3394e3647090d952109f8a543df8d0a31439bf36bc3a506daf6bac97c90a36cb not found: ID does not exist"
Sep 16 18:49:50.510652 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.510624    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"]
Sep 16 18:49:50.533304 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:50.533278    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-af5e21-openshift-default-8dc8b5fb-spm6s"]
Sep 16 18:49:51.419181 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:51.419150    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 18:49:51.419639 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:51.419237    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 18:49:51.784052 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:49:51.784017    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ccf0514-4996-492c-b238-40c982aefa41" path="/var/lib/kubelet/pods/4ccf0514-4996-492c-b238-40c982aefa41/volumes"
Sep 16 18:50:02.032505 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:02.032405    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:50:05.038280 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.038206    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9" podUID="a515228d-d41c-48e0-932c-397d72b4be57" containerName="main" containerID="cri-o://57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d" gracePeriod=45
Sep 16 18:50:05.291425 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.291370    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:50:05.400003 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.399974    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx9sk\" (UniqueName: \"kubernetes.io/projected/a515228d-d41c-48e0-932c-397d72b4be57-kube-api-access-hx9sk\") pod \"a515228d-d41c-48e0-932c-397d72b4be57\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") "
Sep 16 18:50:05.400171 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.400054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-home\") pod \"a515228d-d41c-48e0-932c-397d72b4be57\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") "
Sep 16 18:50:05.400171 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.400090    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-dshm\") pod \"a515228d-d41c-48e0-932c-397d72b4be57\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") "
Sep 16 18:50:05.400171 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.400152    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-tmp-dir\") pod \"a515228d-d41c-48e0-932c-397d72b4be57\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") "
Sep 16 18:50:05.400357 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.400181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-model-cache\") pod \"a515228d-d41c-48e0-932c-397d72b4be57\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") "
Sep 16 18:50:05.400357 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.400258    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a515228d-d41c-48e0-932c-397d72b4be57-tls-certs\") pod \"a515228d-d41c-48e0-932c-397d72b4be57\" (UID: \"a515228d-d41c-48e0-932c-397d72b4be57\") "
Sep 16 18:50:05.400357 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.400318    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-home" (OuterVolumeSpecName: "home") pod "a515228d-d41c-48e0-932c-397d72b4be57" (UID: "a515228d-d41c-48e0-932c-397d72b4be57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:05.400542 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.400433    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a515228d-d41c-48e0-932c-397d72b4be57" (UID: "a515228d-d41c-48e0-932c-397d72b4be57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:05.400542 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.400469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-model-cache" (OuterVolumeSpecName: "model-cache") pod "a515228d-d41c-48e0-932c-397d72b4be57" (UID: "a515228d-d41c-48e0-932c-397d72b4be57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:05.400722 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.400700    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-home\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:50:05.400795 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.400725    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-tmp-dir\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:50:05.400795 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.400736    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-model-cache\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:50:05.402258 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.402224    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-dshm" (OuterVolumeSpecName: "dshm") pod "a515228d-d41c-48e0-932c-397d72b4be57" (UID: "a515228d-d41c-48e0-932c-397d72b4be57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:05.402258 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.402247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a515228d-d41c-48e0-932c-397d72b4be57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a515228d-d41c-48e0-932c-397d72b4be57" (UID: "a515228d-d41c-48e0-932c-397d72b4be57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:05.402414 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.402313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a515228d-d41c-48e0-932c-397d72b4be57-kube-api-access-hx9sk" (OuterVolumeSpecName: "kube-api-access-hx9sk") pod "a515228d-d41c-48e0-932c-397d72b4be57" (UID: "a515228d-d41c-48e0-932c-397d72b4be57"). InnerVolumeSpecName "kube-api-access-hx9sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:05.501466 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.501436    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx9sk\" (UniqueName: \"kubernetes.io/projected/a515228d-d41c-48e0-932c-397d72b4be57-kube-api-access-hx9sk\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:50:05.501466 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.501464    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a515228d-d41c-48e0-932c-397d72b4be57-dshm\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:50:05.501647 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.501478    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a515228d-d41c-48e0-932c-397d72b4be57-tls-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:50:05.555641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.555550    2580 generic.go:356] "Generic (PLEG): container finished" podID="a515228d-d41c-48e0-932c-397d72b4be57" containerID="57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d" exitCode=0
Sep 16 18:50:05.555641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.555598    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"
Sep 16 18:50:05.555641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.555620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9" event={"ID":"a515228d-d41c-48e0-932c-397d72b4be57","Type":"ContainerDied","Data":"57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d"}
Sep 16 18:50:05.555900 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.555653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9" event={"ID":"a515228d-d41c-48e0-932c-397d72b4be57","Type":"ContainerDied","Data":"a75dfb05042d7bacd6193866f9b6fee7b6f6c3bdd38830fe97f089a0b3724d8b"}
Sep 16 18:50:05.555900 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.555669    2580 scope.go:117] "RemoveContainer" containerID="57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d"
Sep 16 18:50:05.565836 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.565817    2580 scope.go:117] "RemoveContainer" containerID="57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d"
Sep 16 18:50:05.566116 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:50:05.566094    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d\": container with ID starting with 57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d not found: ID does not exist" containerID="57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d"
Sep 16 18:50:05.566201 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.566128    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d"} err="failed to get container status \"57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d\": rpc error: code = NotFound desc = could not find container \"57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d\": container with ID starting with 57c029dc23e017a3573087dce0627f22ee9552fcc4946227e5926f81bb36ea2d not found: ID does not exist"
Sep 16 18:50:05.580702 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.580678    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"]
Sep 16 18:50:05.585972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.585953    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-af5e21/noten-model-af5e21-kserve-59cf955c9f-5glm9"]
Sep 16 18:50:05.783675 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:05.783638    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a515228d-d41c-48e0-932c-397d72b4be57" path="/var/lib/kubelet/pods/a515228d-d41c-48e0-932c-397d72b4be57/volumes"
Sep 16 18:50:35.534689 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:35.534656    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:50:40.545320 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.545284    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"]
Sep 16 18:50:40.545714 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.545698    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ccf0514-4996-492c-b238-40c982aefa41" containerName="istio-proxy"
Sep 16 18:50:40.545714 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.545711    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ccf0514-4996-492c-b238-40c982aefa41" containerName="istio-proxy"
Sep 16 18:50:40.545790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.545721    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a515228d-d41c-48e0-932c-397d72b4be57" containerName="main"
Sep 16 18:50:40.545790 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.545726    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a515228d-d41c-48e0-932c-397d72b4be57" containerName="main"
Sep 16 18:50:40.545857 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.545813    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ccf0514-4996-492c-b238-40c982aefa41" containerName="istio-proxy"
Sep 16 18:50:40.545857 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.545825    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a515228d-d41c-48e0-932c-397d72b4be57" containerName="main"
Sep 16 18:50:40.550279 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.550260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"
Sep 16 18:50:40.552778 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.552759    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:50:40.552842 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.552776    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:50:40.566350 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.563651    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"]
Sep 16 18:50:40.624001 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.623967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1641f86e-5012-4da8-8a67-c607b65810f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7\" (UID: \"1641f86e-5012-4da8-8a67-c607b65810f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"
Sep 16 18:50:40.724957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.724922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1641f86e-5012-4da8-8a67-c607b65810f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7\" (UID: \"1641f86e-5012-4da8-8a67-c607b65810f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"
Sep 16 18:50:40.725579 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.725560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1641f86e-5012-4da8-8a67-c607b65810f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7\" (UID: \"1641f86e-5012-4da8-8a67-c607b65810f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"
Sep 16 18:50:40.861054 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:40.860972    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"
Sep 16 18:50:41.134989 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:41.134963    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"]
Sep 16 18:50:41.137479 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:50:41.137452    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1641f86e_5012_4da8_8a67_c607b65810f0.slice/crio-c87d8f5832117890440e5ecfb1fbb703e08df10c7cd5e145dd68cb9db5451ffb WatchSource:0}: Error finding container c87d8f5832117890440e5ecfb1fbb703e08df10c7cd5e145dd68cb9db5451ffb: Status 404 returned error can't find the container with id c87d8f5832117890440e5ecfb1fbb703e08df10c7cd5e145dd68cb9db5451ffb
Sep 16 18:50:41.765284 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:41.765196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7" event={"ID":"1641f86e-5012-4da8-8a67-c607b65810f0","Type":"ContainerStarted","Data":"d6b756b9721227f434b9693e77994cd773aa29373db77bf873983a86ad8f658b"}
Sep 16 18:50:41.765284 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:41.765266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7" event={"ID":"1641f86e-5012-4da8-8a67-c607b65810f0","Type":"ContainerStarted","Data":"c87d8f5832117890440e5ecfb1fbb703e08df10c7cd5e145dd68cb9db5451ffb"}
Sep 16 18:50:41.788957 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:41.788910    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7" podStartSLOduration=1.788894876 podStartE2EDuration="1.788894876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:41.788323335 +0000 UTC m=+1384.497562727" watchObservedRunningTime="2026-09-16 18:50:41.788894876 +0000 UTC m=+1384.498134280"
Sep 16 18:50:42.772424 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:42.772388    2580 generic.go:356] "Generic (PLEG): container finished" podID="1641f86e-5012-4da8-8a67-c607b65810f0" containerID="d6b756b9721227f434b9693e77994cd773aa29373db77bf873983a86ad8f658b" exitCode=0
Sep 16 18:50:42.772800 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:42.772431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7" event={"ID":"1641f86e-5012-4da8-8a67-c607b65810f0","Type":"ContainerDied","Data":"d6b756b9721227f434b9693e77994cd773aa29373db77bf873983a86ad8f658b"}
Sep 16 18:50:43.912711 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:43.912686    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"
Sep 16 18:50:43.959443 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:43.959404    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1641f86e-5012-4da8-8a67-c607b65810f0-maas-api-service-ca\") pod \"1641f86e-5012-4da8-8a67-c607b65810f0\" (UID: \"1641f86e-5012-4da8-8a67-c607b65810f0\") "
Sep 16 18:50:43.959748 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:43.959726    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1641f86e-5012-4da8-8a67-c607b65810f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1641f86e-5012-4da8-8a67-c607b65810f0" (UID: "1641f86e-5012-4da8-8a67-c607b65810f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:43.960060 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:43.959918    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1641f86e-5012-4da8-8a67-c607b65810f0-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:50:44.784384 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:44.784325    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"
Sep 16 18:50:44.784573 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:44.784320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7" event={"ID":"1641f86e-5012-4da8-8a67-c607b65810f0","Type":"ContainerDied","Data":"c87d8f5832117890440e5ecfb1fbb703e08df10c7cd5e145dd68cb9db5451ffb"}
Sep 16 18:50:44.784573 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:50:44.784486    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c87d8f5832117890440e5ecfb1fbb703e08df10c7cd5e145dd68cb9db5451ffb"
Sep 16 18:51:06.652108 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:51:06.652061    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"]
Sep 16 18:51:06.655126 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:51:06.655099    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d91d95b4-5dmq7"]
Sep 16 18:51:07.783240 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:51:07.783205    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1641f86e-5012-4da8-8a67-c607b65810f0" path="/var/lib/kubelet/pods/1641f86e-5012-4da8-8a67-c607b65810f0/volumes"
Sep 16 18:51:08.721859 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:51:08.721821    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:51:38.026413 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:51:38.026382    2580 scope.go:117] "RemoveContainer" containerID="44c7923b7556f7643f001e613fe4ce52310b3093730700ca1bad1e33675a1906"
Sep 16 18:51:38.034839 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:51:38.034817    2580 scope.go:117] "RemoveContainer" containerID="cde9d301ef5ebea3005175b7ad9a8629144d3fcd1f85b1915e21550e1836323e"
Sep 16 18:51:40.059644 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:51:40.059606    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:52:05.332617 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:05.332586    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:52:28.496444 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:28.496409    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"]
Sep 16 18:52:28.496923 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:28.496625    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy" containerID="cri-o://148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a" gracePeriod=30
Sep 16 18:52:33.142651 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.142611    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:52:33.479456 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.479362    2580 patch_prober.go:28] interesting pod/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:52:33.479615 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.479442    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:33.751615 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.751588    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:52:33.847787 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.847755    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8mw9\" (UniqueName: \"kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-kube-api-access-l8mw9\") pod \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") "
Sep 16 18:52:33.847975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.847816    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-podinfo\") pod \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") "
Sep 16 18:52:33.847975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.847849    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istiod-ca-cert\") pod \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") "
Sep 16 18:52:33.847975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.847886    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-certs\") pod \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") "
Sep 16 18:52:33.847975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.847911    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-envoy\") pod \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") "
Sep 16 18:52:33.847975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.847948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-token\") pod \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") "
Sep 16 18:52:33.847975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.847972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-credential-socket\") pod \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") "
Sep 16 18:52:33.848318 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.847996    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-data\") pod \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") "
Sep 16 18:52:33.848318 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.848073    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-socket\") pod \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\" (UID: \"7eb74c06-dff3-4c09-a7e0-cf2a86e74937\") "
Sep 16 18:52:33.848468 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.848426    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7eb74c06-dff3-4c09-a7e0-cf2a86e74937" (UID: "7eb74c06-dff3-4c09-a7e0-cf2a86e74937"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:33.848650 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.848604    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7eb74c06-dff3-4c09-a7e0-cf2a86e74937" (UID: "7eb74c06-dff3-4c09-a7e0-cf2a86e74937"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:33.848742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.848666    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-data" (OuterVolumeSpecName: "istio-data") pod "7eb74c06-dff3-4c09-a7e0-cf2a86e74937" (UID: "7eb74c06-dff3-4c09-a7e0-cf2a86e74937"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:33.848742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.848678    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7eb74c06-dff3-4c09-a7e0-cf2a86e74937" (UID: "7eb74c06-dff3-4c09-a7e0-cf2a86e74937"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:33.848865 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.848715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7eb74c06-dff3-4c09-a7e0-cf2a86e74937" (UID: "7eb74c06-dff3-4c09-a7e0-cf2a86e74937"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:33.850672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.850646    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-kube-api-access-l8mw9" (OuterVolumeSpecName: "kube-api-access-l8mw9") pod "7eb74c06-dff3-4c09-a7e0-cf2a86e74937" (UID: "7eb74c06-dff3-4c09-a7e0-cf2a86e74937"). InnerVolumeSpecName "kube-api-access-l8mw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:33.851064 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.851029    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7eb74c06-dff3-4c09-a7e0-cf2a86e74937" (UID: "7eb74c06-dff3-4c09-a7e0-cf2a86e74937"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:52:33.851209 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.851187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-token" (OuterVolumeSpecName: "istio-token") pod "7eb74c06-dff3-4c09-a7e0-cf2a86e74937" (UID: "7eb74c06-dff3-4c09-a7e0-cf2a86e74937"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:33.851285 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.851225    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7eb74c06-dff3-4c09-a7e0-cf2a86e74937" (UID: "7eb74c06-dff3-4c09-a7e0-cf2a86e74937"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:33.949013 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.948977    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:52:33.949013 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.949007    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8mw9\" (UniqueName: \"kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-kube-api-access-l8mw9\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:52:33.949013 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.949018    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-podinfo\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:52:33.949236 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.949027    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istiod-ca-cert\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:52:33.949236 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.949036    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-workload-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:52:33.949236 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.949045    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-envoy\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:52:33.949236 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.949053    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-token\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:52:33.949236 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.949061    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-credential-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:52:33.949236 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:33.949069    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb74c06-dff3-4c09-a7e0-cf2a86e74937-istio-data\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:52:34.407661 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.407620    2580 generic.go:356] "Generic (PLEG): container finished" podID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerID="148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a" exitCode=0
Sep 16 18:52:34.408076 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.407684    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"
Sep 16 18:52:34.408076 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.407706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" event={"ID":"7eb74c06-dff3-4c09-a7e0-cf2a86e74937","Type":"ContainerDied","Data":"148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a"}
Sep 16 18:52:34.408076 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.407766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng" event={"ID":"7eb74c06-dff3-4c09-a7e0-cf2a86e74937","Type":"ContainerDied","Data":"f09cadf3977f642a970354d72dc02bf1605ce5b95a9e4f5a25b1d9c1b598bd2d"}
Sep 16 18:52:34.408076 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.407792    2580 scope.go:117] "RemoveContainer" containerID="148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a"
Sep 16 18:52:34.418758 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.418739    2580 scope.go:117] "RemoveContainer" containerID="82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8"
Sep 16 18:52:34.426962 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.426947    2580 scope.go:117] "RemoveContainer" containerID="148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a"
Sep 16 18:52:34.427222 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:52:34.427197    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a\": container with ID starting with 148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a not found: ID does not exist" containerID="148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a"
Sep 16 18:52:34.427312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.427228    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a"} err="failed to get container status \"148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a\": rpc error: code = NotFound desc = could not find container \"148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a\": container with ID starting with 148cb722828f48b41baa741e829555a7decb61144500b7d3f1dbcf4677339a2a not found: ID does not exist"
Sep 16 18:52:34.427312 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.427252    2580 scope.go:117] "RemoveContainer" containerID="82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8"
Sep 16 18:52:34.427550 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:52:34.427530    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8\": container with ID starting with 82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8 not found: ID does not exist" containerID="82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8"
Sep 16 18:52:34.427591 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.427556    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8"} err="failed to get container status \"82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8\": rpc error: code = NotFound desc = could not find container \"82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8\": container with ID starting with 82a3d0f2518872b44d09d11820cfbb2af3f519d46e61e11fd969184949c2c8e8 not found: ID does not exist"
Sep 16 18:52:34.432780 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.432757    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"]
Sep 16 18:52:34.438321 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:34.438302    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d91d95b4-openshift-default-c7c99c4cb-zh6ng"]
Sep 16 18:52:35.784197 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:35.784163    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" path="/var/lib/kubelet/pods/7eb74c06-dff3-4c09-a7e0-cf2a86e74937/volumes"
Sep 16 18:52:37.850940 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:37.850908    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:52:37.853558 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:37.853535    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:52:38.062256 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:38.062225    2580 scope.go:117] "RemoveContainer" containerID="1104dcb279baf9caa68bd800fc737e6c9447a9f18622007d04e5a0a2aa602a15"
Sep 16 18:52:45.278008 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.277976    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"]
Sep 16 18:52:45.278402 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.278386    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1641f86e-5012-4da8-8a67-c607b65810f0" containerName="revoke-keys"
Sep 16 18:52:45.278402 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.278399    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1641f86e-5012-4da8-8a67-c607b65810f0" containerName="revoke-keys"
Sep 16 18:52:45.278481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.278412    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy"
Sep 16 18:52:45.278481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.278418    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy"
Sep 16 18:52:45.278481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.278428    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy"
Sep 16 18:52:45.278481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.278433    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy"
Sep 16 18:52:45.278606 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.278503    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1641f86e-5012-4da8-8a67-c607b65810f0" containerName="revoke-keys"
Sep 16 18:52:45.278606 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.278511    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy"
Sep 16 18:52:45.278606 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.278518    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb74c06-dff3-4c09-a7e0-cf2a86e74937" containerName="istio-proxy"
Sep 16 18:52:45.281768 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.281752    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.284278 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.284254    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-662432\"/\"distinct-2-w4-662432-kserve-self-signed-certs\""
Sep 16 18:52:45.306322 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.306298    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"]
Sep 16 18:52:45.358407 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.358380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-dshm\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.358541 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.358429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-model-cache\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.358541 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.358451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tmp-dir\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.358541 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.358516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-home\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.358678 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.358552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tls-certs\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.358678 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.358586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6t95\" (UniqueName: \"kubernetes.io/projected/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-kube-api-access-k6t95\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.459959 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.459926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-model-cache\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.459959 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.459959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tmp-dir\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.460214 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.459979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-home\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.460214 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.459998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tls-certs\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.460214 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.460188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6t95\" (UniqueName: \"kubernetes.io/projected/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-kube-api-access-k6t95\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.460475 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.460366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-dshm\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.460475 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.460422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-model-cache\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.460603 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.460536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tmp-dir\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.460731 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.460705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-home\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.462503 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.462483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-dshm\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.462966 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.462949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tls-certs\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.469301 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.469280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6t95\" (UniqueName: \"kubernetes.io/projected/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-kube-api-access-k6t95\") pod \"distinct-2-w4-662432-kserve-7789d6dd47-s6q9b\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") " pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.590733 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.590653    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:45.865889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:45.865864    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"]
Sep 16 18:52:45.868343 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:52:45.868303    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf10b5732_a5d0_474a_8fbb_7ae3727c60bc.slice/crio-612c659ad47bfa0b37e9e97c3780730db507f6d57df5bbc39dcb88b4d5c89f31 WatchSource:0}: Error finding container 612c659ad47bfa0b37e9e97c3780730db507f6d57df5bbc39dcb88b4d5c89f31: Status 404 returned error can't find the container with id 612c659ad47bfa0b37e9e97c3780730db507f6d57df5bbc39dcb88b4d5c89f31
Sep 16 18:52:46.483704 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:46.483650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b" event={"ID":"f10b5732-a5d0-474a-8fbb-7ae3727c60bc","Type":"ContainerStarted","Data":"416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a"}
Sep 16 18:52:46.484137 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:46.483716    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b" event={"ID":"f10b5732-a5d0-474a-8fbb-7ae3727c60bc","Type":"ContainerStarted","Data":"612c659ad47bfa0b37e9e97c3780730db507f6d57df5bbc39dcb88b4d5c89f31"}
Sep 16 18:52:46.516351 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:46.516281    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b" podStartSLOduration=1.516264635 podStartE2EDuration="1.516264635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:46.513912865 +0000 UTC m=+1509.223152254" watchObservedRunningTime="2026-09-16 18:52:46.516264635 +0000 UTC m=+1509.225504026"
Sep 16 18:52:55.591268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:55.591236    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:55.591716 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:55.591282    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:55.603704 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:55.603673    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:52:56.549801 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:52:56.549768    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:53:06.847500 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:06.847422    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:53:36.357284 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.357248    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"]
Sep 16 18:53:36.360960 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.360934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:53:36.363410 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.363381    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-66b561-dockercfg-cd85n\""
Sep 16 18:53:36.363563 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.363537    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-66b561\""
Sep 16 18:53:36.371578 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.371554    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"]
Sep 16 18:53:36.388475 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.388448    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"]
Sep 16 18:53:36.391926 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.391907    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:53:36.400871 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.400847    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"]
Sep 16 18:53:36.481268 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.481235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k4tw\" (UniqueName: \"kubernetes.io/projected/311b3b86-5413-4a4c-885a-fea1e3ffe045-kube-api-access-2k4tw\") pod \"payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s\" (UID: \"311b3b86-5413-4a4c-885a-fea1e3ffe045\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:53:36.481466 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.481299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48c6f1b-4930-45ed-94f1-12428445d484-plugins-config-volume\") pod \"payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx\" (UID: \"b48c6f1b-4930-45ed-94f1-12428445d484\") " pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:53:36.481466 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.481372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4xc\" (UniqueName: \"kubernetes.io/projected/b48c6f1b-4930-45ed-94f1-12428445d484-kube-api-access-kb4xc\") pod \"payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx\" (UID: \"b48c6f1b-4930-45ed-94f1-12428445d484\") " pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:53:36.481578 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.481482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311b3b86-5413-4a4c-885a-fea1e3ffe045-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s\" (UID: \"311b3b86-5413-4a4c-885a-fea1e3ffe045\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:53:36.583001 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.582960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311b3b86-5413-4a4c-885a-fea1e3ffe045-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s\" (UID: \"311b3b86-5413-4a4c-885a-fea1e3ffe045\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:53:36.583172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.583048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k4tw\" (UniqueName: \"kubernetes.io/projected/311b3b86-5413-4a4c-885a-fea1e3ffe045-kube-api-access-2k4tw\") pod \"payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s\" (UID: \"311b3b86-5413-4a4c-885a-fea1e3ffe045\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:53:36.583172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.583113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48c6f1b-4930-45ed-94f1-12428445d484-plugins-config-volume\") pod \"payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx\" (UID: \"b48c6f1b-4930-45ed-94f1-12428445d484\") " pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:53:36.583172 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.583146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4xc\" (UniqueName: \"kubernetes.io/projected/b48c6f1b-4930-45ed-94f1-12428445d484-kube-api-access-kb4xc\") pod \"payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx\" (UID: \"b48c6f1b-4930-45ed-94f1-12428445d484\") " pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:53:36.583829 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.583810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311b3b86-5413-4a4c-885a-fea1e3ffe045-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s\" (UID: \"311b3b86-5413-4a4c-885a-fea1e3ffe045\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:53:36.583875 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.583857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48c6f1b-4930-45ed-94f1-12428445d484-plugins-config-volume\") pod \"payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx\" (UID: \"b48c6f1b-4930-45ed-94f1-12428445d484\") " pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:53:36.592511 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.592486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k4tw\" (UniqueName: \"kubernetes.io/projected/311b3b86-5413-4a4c-885a-fea1e3ffe045-kube-api-access-2k4tw\") pod \"payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s\" (UID: \"311b3b86-5413-4a4c-885a-fea1e3ffe045\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:53:36.592626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.592609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4xc\" (UniqueName: \"kubernetes.io/projected/b48c6f1b-4930-45ed-94f1-12428445d484-kube-api-access-kb4xc\") pod \"payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx\" (UID: \"b48c6f1b-4930-45ed-94f1-12428445d484\") " pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:53:36.674193 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.674117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:53:36.703690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.703663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:53:36.951467 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.951179    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"]
Sep 16 18:53:36.955076 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:53:36.955048    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311b3b86_5413_4a4c_885a_fea1e3ffe045.slice/crio-2802c5c2c685e524791c196ecc7086969b2238f32f4eb4bb7bb9ab5788e19287 WatchSource:0}: Error finding container 2802c5c2c685e524791c196ecc7086969b2238f32f4eb4bb7bb9ab5788e19287: Status 404 returned error can't find the container with id 2802c5c2c685e524791c196ecc7086969b2238f32f4eb4bb7bb9ab5788e19287
Sep 16 18:53:36.988634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:36.988603    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"]
Sep 16 18:53:36.990368 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:53:36.990319    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48c6f1b_4930_45ed_94f1_12428445d484.slice/crio-9bb4cadccb78efc2972716c6027984983745ebcce1637819e650e15ae25d6d29 WatchSource:0}: Error finding container 9bb4cadccb78efc2972716c6027984983745ebcce1637819e650e15ae25d6d29: Status 404 returned error can't find the container with id 9bb4cadccb78efc2972716c6027984983745ebcce1637819e650e15ae25d6d29
Sep 16 18:53:37.433709 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:37.433664    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:53:37.784704 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:37.784674    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:53:37.784893 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:37.784721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s" event={"ID":"311b3b86-5413-4a4c-885a-fea1e3ffe045","Type":"ContainerStarted","Data":"841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda"}
Sep 16 18:53:37.784893 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:37.784748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s" event={"ID":"311b3b86-5413-4a4c-885a-fea1e3ffe045","Type":"ContainerStarted","Data":"2802c5c2c685e524791c196ecc7086969b2238f32f4eb4bb7bb9ab5788e19287"}
Sep 16 18:53:37.784893 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:37.784762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx" event={"ID":"b48c6f1b-4930-45ed-94f1-12428445d484","Type":"ContainerStarted","Data":"2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d"}
Sep 16 18:53:37.784893 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:37.784779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx" event={"ID":"b48c6f1b-4930-45ed-94f1-12428445d484","Type":"ContainerStarted","Data":"9bb4cadccb78efc2972716c6027984983745ebcce1637819e650e15ae25d6d29"}
Sep 16 18:53:37.799364 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:37.799150    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s" podStartSLOduration=1.799128281 podStartE2EDuration="1.799128281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:37.797592522 +0000 UTC m=+1560.506831916" watchObservedRunningTime="2026-09-16 18:53:37.799128281 +0000 UTC m=+1560.508367679"
Sep 16 18:53:37.818229 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:37.818167    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx" podStartSLOduration=1.8181465700000001 podStartE2EDuration="1.81814657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:37.817346635 +0000 UTC m=+1560.526586019" watchObservedRunningTime="2026-09-16 18:53:37.81814657 +0000 UTC m=+1560.527385962"
Sep 16 18:53:38.087364 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:38.087268    2580 scope.go:117] "RemoveContainer" containerID="5c2f4aea64793250e61e674971c7d5c592451a82dfdd99c52e654bb673b70d1d"
Sep 16 18:53:38.788057 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:38.788024    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:53:48.790316 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:48.790284    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:53:49.793706 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:53:49.793676    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:54:01.768518 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:01.768479    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"]
Sep 16 18:54:01.772221 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:01.772201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:54:01.774780 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:01.774759    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-41abff\""
Sep 16 18:54:01.774867 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:01.774778    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-41abff-dockercfg-xj8kj\""
Sep 16 18:54:01.784860 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:01.784834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"]
Sep 16 18:54:01.847634 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:01.847600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8768ed75-5fea-4a32-84c1-22f0e6512de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp\" (UID: \"8768ed75-5fea-4a32-84c1-22f0e6512de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:54:01.847849 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:01.847743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdcwd\" (UniqueName: \"kubernetes.io/projected/8768ed75-5fea-4a32-84c1-22f0e6512de1-kube-api-access-fdcwd\") pod \"payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp\" (UID: \"8768ed75-5fea-4a32-84c1-22f0e6512de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:54:01.950197 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:01.950151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdcwd\" (UniqueName: \"kubernetes.io/projected/8768ed75-5fea-4a32-84c1-22f0e6512de1-kube-api-access-fdcwd\") pod \"payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp\" (UID: \"8768ed75-5fea-4a32-84c1-22f0e6512de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:54:01.950433 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:01.950321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8768ed75-5fea-4a32-84c1-22f0e6512de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp\" (UID: \"8768ed75-5fea-4a32-84c1-22f0e6512de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:54:01.951294 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:01.951260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8768ed75-5fea-4a32-84c1-22f0e6512de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp\" (UID: \"8768ed75-5fea-4a32-84c1-22f0e6512de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:54:01.961719 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:01.961685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdcwd\" (UniqueName: \"kubernetes.io/projected/8768ed75-5fea-4a32-84c1-22f0e6512de1-kube-api-access-fdcwd\") pod \"payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp\" (UID: \"8768ed75-5fea-4a32-84c1-22f0e6512de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:54:02.083428 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:02.083322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:54:02.358994 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:02.358968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"]
Sep 16 18:54:02.360961 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:54:02.360933    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8768ed75_5fea_4a32_84c1_22f0e6512de1.slice/crio-3bc528cd3a19f201df2b630cbffc549052065bf503edb6b52ded2892fda6cea1 WatchSource:0}: Error finding container 3bc528cd3a19f201df2b630cbffc549052065bf503edb6b52ded2892fda6cea1: Status 404 returned error can't find the container with id 3bc528cd3a19f201df2b630cbffc549052065bf503edb6b52ded2892fda6cea1
Sep 16 18:54:02.926263 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:02.926230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp" event={"ID":"8768ed75-5fea-4a32-84c1-22f0e6512de1","Type":"ContainerStarted","Data":"2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4"}
Sep 16 18:54:02.926667 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:02.926287    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:54:02.926667 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:02.926307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp" event={"ID":"8768ed75-5fea-4a32-84c1-22f0e6512de1","Type":"ContainerStarted","Data":"3bc528cd3a19f201df2b630cbffc549052065bf503edb6b52ded2892fda6cea1"}
Sep 16 18:54:02.950241 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:02.950187    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp" podStartSLOduration=1.950170132 podStartE2EDuration="1.950170132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:02.947993855 +0000 UTC m=+1585.657233277" watchObservedRunningTime="2026-09-16 18:54:02.950170132 +0000 UTC m=+1585.659409521"
Sep 16 18:54:13.934781 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:13.934742    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:54:17.858079 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:17.858049    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"]
Sep 16 18:54:17.861547 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:17.861513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"
Sep 16 18:54:17.863723 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:17.863698    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:54:17.863848 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:17.863754    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:54:17.871408 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:17.871377    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"]
Sep 16 18:54:17.911069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:17.911038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9829f6ff-632e-4295-9e40-34dcb4f182fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-66b561-crttg\" (UID: \"9829f6ff-632e-4295-9e40-34dcb4f182fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"
Sep 16 18:54:18.011915 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:18.011876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9829f6ff-632e-4295-9e40-34dcb4f182fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-66b561-crttg\" (UID: \"9829f6ff-632e-4295-9e40-34dcb4f182fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"
Sep 16 18:54:18.012574 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:18.012553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9829f6ff-632e-4295-9e40-34dcb4f182fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-66b561-crttg\" (UID: \"9829f6ff-632e-4295-9e40-34dcb4f182fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"
Sep 16 18:54:18.173709 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:18.173629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"
Sep 16 18:54:18.444561 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:18.444535    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"]
Sep 16 18:54:19.024374 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:19.024271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg" event={"ID":"9829f6ff-632e-4295-9e40-34dcb4f182fe","Type":"ContainerStarted","Data":"fcb92e19e7aa205167df0e267ccb7b4aa4b3398cb96bf534a32ac61c0df02464"}
Sep 16 18:54:19.024374 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:19.024368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg" event={"ID":"9829f6ff-632e-4295-9e40-34dcb4f182fe","Type":"ContainerStarted","Data":"35286058449f059761c7a757d12e5bedc9a0934dd51d428803ea2fdbd6a019f7"}
Sep 16 18:54:19.042035 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:19.041984    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg" podStartSLOduration=2.041969455 podStartE2EDuration="2.041969455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:19.040549275 +0000 UTC m=+1601.749788669" watchObservedRunningTime="2026-09-16 18:54:19.041969455 +0000 UTC m=+1601.751208848"
Sep 16 18:54:20.031341 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:20.031304    2580 generic.go:356] "Generic (PLEG): container finished" podID="9829f6ff-632e-4295-9e40-34dcb4f182fe" containerID="fcb92e19e7aa205167df0e267ccb7b4aa4b3398cb96bf534a32ac61c0df02464" exitCode=0
Sep 16 18:54:20.031785 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:20.031387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg" event={"ID":"9829f6ff-632e-4295-9e40-34dcb4f182fe","Type":"ContainerDied","Data":"fcb92e19e7aa205167df0e267ccb7b4aa4b3398cb96bf534a32ac61c0df02464"}
Sep 16 18:54:21.185352 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:21.185318    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"
Sep 16 18:54:21.241785 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:21.241755    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9829f6ff-632e-4295-9e40-34dcb4f182fe-maas-api-service-ca\") pod \"9829f6ff-632e-4295-9e40-34dcb4f182fe\" (UID: \"9829f6ff-632e-4295-9e40-34dcb4f182fe\") "
Sep 16 18:54:21.242088 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:21.242066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9829f6ff-632e-4295-9e40-34dcb4f182fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9829f6ff-632e-4295-9e40-34dcb4f182fe" (UID: "9829f6ff-632e-4295-9e40-34dcb4f182fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:21.343529 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:21.343449    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9829f6ff-632e-4295-9e40-34dcb4f182fe-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:54:22.042875 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:22.042791    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"
Sep 16 18:54:22.042875 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:22.042795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg" event={"ID":"9829f6ff-632e-4295-9e40-34dcb4f182fe","Type":"ContainerDied","Data":"35286058449f059761c7a757d12e5bedc9a0934dd51d428803ea2fdbd6a019f7"}
Sep 16 18:54:22.042875 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:22.042851    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35286058449f059761c7a757d12e5bedc9a0934dd51d428803ea2fdbd6a019f7"
Sep 16 18:54:24.624922 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:24.624887    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:54:38.107978 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:38.107947    2580 scope.go:117] "RemoveContainer" containerID="86036c7bd617339d4e0da7b35286bce16875ae27ca6ac7429f003890c3b3ac9e"
Sep 16 18:54:38.116579 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:38.116562    2580 scope.go:117] "RemoveContainer" containerID="c04c81a0a1e2b04d8ec70e73000e29628db7e0be7a8c774ef35ea81511459a54"
Sep 16 18:54:39.540901 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.540863    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"]
Sep 16 18:54:39.541296 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.541110    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx" podUID="b48c6f1b-4930-45ed-94f1-12428445d484" containerName="payload-processing" containerID="cri-o://2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d" gracePeriod=30
Sep 16 18:54:39.545812 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.545786    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"]
Sep 16 18:54:39.546045 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.546000    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s" podUID="311b3b86-5413-4a4c-885a-fea1e3ffe045" containerName="payload-pre-processing" containerID="cri-o://841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda" gracePeriod=30
Sep 16 18:54:39.751730 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.751696    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"]
Sep 16 18:54:39.754692 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.754667    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-66b561-crttg"]
Sep 16 18:54:39.785899 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.785865    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9829f6ff-632e-4295-9e40-34dcb4f182fe" path="/var/lib/kubelet/pods/9829f6ff-632e-4295-9e40-34dcb4f182fe/volumes"
Sep 16 18:54:39.837125 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.837101    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:54:39.840090 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.840071    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:54:39.938582 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.938552    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k4tw\" (UniqueName: \"kubernetes.io/projected/311b3b86-5413-4a4c-885a-fea1e3ffe045-kube-api-access-2k4tw\") pod \"311b3b86-5413-4a4c-885a-fea1e3ffe045\" (UID: \"311b3b86-5413-4a4c-885a-fea1e3ffe045\") "
Sep 16 18:54:39.938757 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.938588    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48c6f1b-4930-45ed-94f1-12428445d484-plugins-config-volume\") pod \"b48c6f1b-4930-45ed-94f1-12428445d484\" (UID: \"b48c6f1b-4930-45ed-94f1-12428445d484\") "
Sep 16 18:54:39.938757 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.938683    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb4xc\" (UniqueName: \"kubernetes.io/projected/b48c6f1b-4930-45ed-94f1-12428445d484-kube-api-access-kb4xc\") pod \"b48c6f1b-4930-45ed-94f1-12428445d484\" (UID: \"b48c6f1b-4930-45ed-94f1-12428445d484\") "
Sep 16 18:54:39.938757 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.938752    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311b3b86-5413-4a4c-885a-fea1e3ffe045-plugins-config-volume\") pod \"311b3b86-5413-4a4c-885a-fea1e3ffe045\" (UID: \"311b3b86-5413-4a4c-885a-fea1e3ffe045\") "
Sep 16 18:54:39.938999 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.938960    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48c6f1b-4930-45ed-94f1-12428445d484-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b48c6f1b-4930-45ed-94f1-12428445d484" (UID: "b48c6f1b-4930-45ed-94f1-12428445d484"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:39.939187 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.939153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311b3b86-5413-4a4c-885a-fea1e3ffe045-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "311b3b86-5413-4a4c-885a-fea1e3ffe045" (UID: "311b3b86-5413-4a4c-885a-fea1e3ffe045"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:39.940868 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.940847    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48c6f1b-4930-45ed-94f1-12428445d484-kube-api-access-kb4xc" (OuterVolumeSpecName: "kube-api-access-kb4xc") pod "b48c6f1b-4930-45ed-94f1-12428445d484" (UID: "b48c6f1b-4930-45ed-94f1-12428445d484"). InnerVolumeSpecName "kube-api-access-kb4xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:39.940961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:39.940873    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311b3b86-5413-4a4c-885a-fea1e3ffe045-kube-api-access-2k4tw" (OuterVolumeSpecName: "kube-api-access-2k4tw") pod "311b3b86-5413-4a4c-885a-fea1e3ffe045" (UID: "311b3b86-5413-4a4c-885a-fea1e3ffe045"). InnerVolumeSpecName "kube-api-access-2k4tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:40.040459 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.040422    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k4tw\" (UniqueName: \"kubernetes.io/projected/311b3b86-5413-4a4c-885a-fea1e3ffe045-kube-api-access-2k4tw\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:54:40.040459 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.040451    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48c6f1b-4930-45ed-94f1-12428445d484-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:54:40.040459 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.040460    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4xc\" (UniqueName: \"kubernetes.io/projected/b48c6f1b-4930-45ed-94f1-12428445d484-kube-api-access-kb4xc\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:54:40.040670 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.040468    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311b3b86-5413-4a4c-885a-fea1e3ffe045-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:54:40.146731 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.146643    2580 generic.go:356] "Generic (PLEG): container finished" podID="311b3b86-5413-4a4c-885a-fea1e3ffe045" containerID="841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda" exitCode=0
Sep 16 18:54:40.146731 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.146693    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"
Sep 16 18:54:40.146936 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.146730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s" event={"ID":"311b3b86-5413-4a4c-885a-fea1e3ffe045","Type":"ContainerDied","Data":"841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda"}
Sep 16 18:54:40.146936 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.146816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s" event={"ID":"311b3b86-5413-4a4c-885a-fea1e3ffe045","Type":"ContainerDied","Data":"2802c5c2c685e524791c196ecc7086969b2238f32f4eb4bb7bb9ab5788e19287"}
Sep 16 18:54:40.146936 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.146845    2580 scope.go:117] "RemoveContainer" containerID="841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda"
Sep 16 18:54:40.148148 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.148116    2580 generic.go:356] "Generic (PLEG): container finished" podID="b48c6f1b-4930-45ed-94f1-12428445d484" containerID="2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d" exitCode=0
Sep 16 18:54:40.148314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.148153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx" event={"ID":"b48c6f1b-4930-45ed-94f1-12428445d484","Type":"ContainerDied","Data":"2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d"}
Sep 16 18:54:40.148314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.148173    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"
Sep 16 18:54:40.148314 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.148204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx" event={"ID":"b48c6f1b-4930-45ed-94f1-12428445d484","Type":"ContainerDied","Data":"9bb4cadccb78efc2972716c6027984983745ebcce1637819e650e15ae25d6d29"}
Sep 16 18:54:40.156985 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.156966    2580 scope.go:117] "RemoveContainer" containerID="841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda"
Sep 16 18:54:40.157286 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:54:40.157262    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda\": container with ID starting with 841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda not found: ID does not exist" containerID="841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda"
Sep 16 18:54:40.157429 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.157292    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda"} err="failed to get container status \"841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda\": rpc error: code = NotFound desc = could not find container \"841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda\": container with ID starting with 841ec89167714a6174f7ede24a9ee9e307c5c339b909f93eac5b26dbb07c6cda not found: ID does not exist"
Sep 16 18:54:40.157429 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.157311    2580 scope.go:117] "RemoveContainer" containerID="2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d"
Sep 16 18:54:40.167010 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.166995    2580 scope.go:117] "RemoveContainer" containerID="2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d"
Sep 16 18:54:40.167316 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:54:40.167298    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d\": container with ID starting with 2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d not found: ID does not exist" containerID="2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d"
Sep 16 18:54:40.167393 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.167323    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d"} err="failed to get container status \"2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d\": rpc error: code = NotFound desc = could not find container \"2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d\": container with ID starting with 2f323b67033a69e8c5420cc510a45108cc641ed14688a1279d9383f637d4451d not found: ID does not exist"
Sep 16 18:54:40.171925 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.171905    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"]
Sep 16 18:54:40.176431 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.176408    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s"]
Sep 16 18:54:40.188299 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.188272    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"]
Sep 16 18:54:40.194017 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.193997    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-66b561-777c4cf599-gdgkx"]
Sep 16 18:54:40.793431 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:40.793388    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-66b561-7f9bc4f69d-rts9s" podUID="311b3b86-5413-4a4c-885a-fea1e3ffe045" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.84:9004: i/o timeout"
Sep 16 18:54:41.783986 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:41.783946    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311b3b86-5413-4a4c-885a-fea1e3ffe045" path="/var/lib/kubelet/pods/311b3b86-5413-4a4c-885a-fea1e3ffe045/volumes"
Sep 16 18:54:41.784466 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:41.784446    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b48c6f1b-4930-45ed-94f1-12428445d484" path="/var/lib/kubelet/pods/b48c6f1b-4930-45ed-94f1-12428445d484/volumes"
Sep 16 18:54:47.581703 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.581667    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"]
Sep 16 18:54:47.582127 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.582078    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48c6f1b-4930-45ed-94f1-12428445d484" containerName="payload-processing"
Sep 16 18:54:47.582127 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.582091    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48c6f1b-4930-45ed-94f1-12428445d484" containerName="payload-processing"
Sep 16 18:54:47.582127 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.582108    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311b3b86-5413-4a4c-885a-fea1e3ffe045" containerName="payload-pre-processing"
Sep 16 18:54:47.582127 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.582113    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="311b3b86-5413-4a4c-885a-fea1e3ffe045" containerName="payload-pre-processing"
Sep 16 18:54:47.582127 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.582120    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9829f6ff-632e-4295-9e40-34dcb4f182fe" containerName="revoke-keys"
Sep 16 18:54:47.582127 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.582126    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9829f6ff-632e-4295-9e40-34dcb4f182fe" containerName="revoke-keys"
Sep 16 18:54:47.582361 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.582185    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="311b3b86-5413-4a4c-885a-fea1e3ffe045" containerName="payload-pre-processing"
Sep 16 18:54:47.582361 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.582194    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48c6f1b-4930-45ed-94f1-12428445d484" containerName="payload-processing"
Sep 16 18:54:47.582361 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.582200    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9829f6ff-632e-4295-9e40-34dcb4f182fe" containerName="revoke-keys"
Sep 16 18:54:47.586659 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.586643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"
Sep 16 18:54:47.589031 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.589009    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:54:47.589133 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.589063    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:54:47.593224 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.593199    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"]
Sep 16 18:54:47.614592 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.614564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d69608-1f15-44d8-a2e8-cd0bc32f341b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-41abff-fxjk8\" (UID: \"42d69608-1f15-44d8-a2e8-cd0bc32f341b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"
Sep 16 18:54:47.715404 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.715373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d69608-1f15-44d8-a2e8-cd0bc32f341b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-41abff-fxjk8\" (UID: \"42d69608-1f15-44d8-a2e8-cd0bc32f341b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"
Sep 16 18:54:47.716000 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.715981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d69608-1f15-44d8-a2e8-cd0bc32f341b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-41abff-fxjk8\" (UID: \"42d69608-1f15-44d8-a2e8-cd0bc32f341b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"
Sep 16 18:54:47.900094 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:47.900016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"
Sep 16 18:54:48.181011 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:48.180815    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"]
Sep 16 18:54:48.183574 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:54:48.183546    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d69608_1f15_44d8_a2e8_cd0bc32f341b.slice/crio-5ca66dd241a163d025a50e4a3ff30de3e3b10aa95b3a21836a060a33fe0e92c8 WatchSource:0}: Error finding container 5ca66dd241a163d025a50e4a3ff30de3e3b10aa95b3a21836a060a33fe0e92c8: Status 404 returned error can't find the container with id 5ca66dd241a163d025a50e4a3ff30de3e3b10aa95b3a21836a060a33fe0e92c8
Sep 16 18:54:48.196843 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:48.196821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8" event={"ID":"42d69608-1f15-44d8-a2e8-cd0bc32f341b","Type":"ContainerStarted","Data":"5ca66dd241a163d025a50e4a3ff30de3e3b10aa95b3a21836a060a33fe0e92c8"}
Sep 16 18:54:48.666273 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:48.666238    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"]
Sep 16 18:54:48.669713 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:48.669693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"
Sep 16 18:54:48.677509 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:48.677447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"]
Sep 16 18:54:48.725998 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:48.725929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209d00ec-8f7e-4e87-b70b-49d129a62ae6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5\" (UID: \"209d00ec-8f7e-4e87-b70b-49d129a62ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"
Sep 16 18:54:48.826540 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:48.826503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209d00ec-8f7e-4e87-b70b-49d129a62ae6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5\" (UID: \"209d00ec-8f7e-4e87-b70b-49d129a62ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"
Sep 16 18:54:48.827184 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:48.827162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209d00ec-8f7e-4e87-b70b-49d129a62ae6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5\" (UID: \"209d00ec-8f7e-4e87-b70b-49d129a62ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"
Sep 16 18:54:48.982483 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:48.982420    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"
Sep 16 18:54:49.203261 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:49.203225    2580 generic.go:356] "Generic (PLEG): container finished" podID="42d69608-1f15-44d8-a2e8-cd0bc32f341b" containerID="9c6b7cba5abd1049077c15ea11b0354695fec618fe47b9ee0167490eca05ec70" exitCode=0
Sep 16 18:54:49.203440 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:49.203319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8" event={"ID":"42d69608-1f15-44d8-a2e8-cd0bc32f341b","Type":"ContainerDied","Data":"9c6b7cba5abd1049077c15ea11b0354695fec618fe47b9ee0167490eca05ec70"}
Sep 16 18:54:49.269526 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:49.269504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"]
Sep 16 18:54:49.271430 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:54:49.271407    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209d00ec_8f7e_4e87_b70b_49d129a62ae6.slice/crio-4e622ccf20a3259d9895c87b1e9e7e53a4c9ed8d06c52e3f7297c7f6e9f514c9 WatchSource:0}: Error finding container 4e622ccf20a3259d9895c87b1e9e7e53a4c9ed8d06c52e3f7297c7f6e9f514c9: Status 404 returned error can't find the container with id 4e622ccf20a3259d9895c87b1e9e7e53a4c9ed8d06c52e3f7297c7f6e9f514c9
Sep 16 18:54:50.210522 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:50.210416    2580 generic.go:356] "Generic (PLEG): container finished" podID="209d00ec-8f7e-4e87-b70b-49d129a62ae6" containerID="4e0d3f59684c2d6bcee1e5c36660cff29c43aab496cf222f85269ae9a9b00a9a" exitCode=0
Sep 16 18:54:50.210522 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:50.210495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5" event={"ID":"209d00ec-8f7e-4e87-b70b-49d129a62ae6","Type":"ContainerDied","Data":"4e0d3f59684c2d6bcee1e5c36660cff29c43aab496cf222f85269ae9a9b00a9a"}
Sep 16 18:54:50.211000 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:50.210538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5" event={"ID":"209d00ec-8f7e-4e87-b70b-49d129a62ae6","Type":"ContainerStarted","Data":"4e622ccf20a3259d9895c87b1e9e7e53a4c9ed8d06c52e3f7297c7f6e9f514c9"}
Sep 16 18:54:50.352904 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:50.352879    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"
Sep 16 18:54:50.443479 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:50.443444    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d69608-1f15-44d8-a2e8-cd0bc32f341b-maas-api-service-ca\") pod \"42d69608-1f15-44d8-a2e8-cd0bc32f341b\" (UID: \"42d69608-1f15-44d8-a2e8-cd0bc32f341b\") "
Sep 16 18:54:50.443782 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:50.443759    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d69608-1f15-44d8-a2e8-cd0bc32f341b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42d69608-1f15-44d8-a2e8-cd0bc32f341b" (UID: "42d69608-1f15-44d8-a2e8-cd0bc32f341b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:50.544605 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:50.544574    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d69608-1f15-44d8-a2e8-cd0bc32f341b-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:54:51.217584 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:51.217559    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"
Sep 16 18:54:51.217996 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:51.217593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8" event={"ID":"42d69608-1f15-44d8-a2e8-cd0bc32f341b","Type":"ContainerDied","Data":"5ca66dd241a163d025a50e4a3ff30de3e3b10aa95b3a21836a060a33fe0e92c8"}
Sep 16 18:54:51.217996 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:51.217642    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ca66dd241a163d025a50e4a3ff30de3e3b10aa95b3a21836a060a33fe0e92c8"
Sep 16 18:54:51.351544 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:51.351518    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"
Sep 16 18:54:51.454241 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:51.454212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209d00ec-8f7e-4e87-b70b-49d129a62ae6-maas-api-service-ca\") pod \"209d00ec-8f7e-4e87-b70b-49d129a62ae6\" (UID: \"209d00ec-8f7e-4e87-b70b-49d129a62ae6\") "
Sep 16 18:54:51.454629 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:51.454609    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209d00ec-8f7e-4e87-b70b-49d129a62ae6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "209d00ec-8f7e-4e87-b70b-49d129a62ae6" (UID: "209d00ec-8f7e-4e87-b70b-49d129a62ae6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:51.555581 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:51.555551    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209d00ec-8f7e-4e87-b70b-49d129a62ae6-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:54:52.224590 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:52.224558    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"
Sep 16 18:54:52.224985 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:52.224599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5" event={"ID":"209d00ec-8f7e-4e87-b70b-49d129a62ae6","Type":"ContainerDied","Data":"4e622ccf20a3259d9895c87b1e9e7e53a4c9ed8d06c52e3f7297c7f6e9f514c9"}
Sep 16 18:54:52.224985 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:52.224647    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e622ccf20a3259d9895c87b1e9e7e53a4c9ed8d06c52e3f7297c7f6e9f514c9"
Sep 16 18:54:53.631690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:53.631657    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"]
Sep 16 18:54:53.662658 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:53.662629    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"]
Sep 16 18:54:56.248346 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:56.248298    2580 generic.go:356] "Generic (PLEG): container finished" podID="92bc46d0-34a8-44aa-97d8-dd13c100d25d" containerID="6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c" exitCode=137
Sep 16 18:54:56.248795 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:56.248379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc" event={"ID":"92bc46d0-34a8-44aa-97d8-dd13c100d25d","Type":"ContainerDied","Data":"6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c"}
Sep 16 18:54:56.248795 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:56.248745    2580 scope.go:117] "RemoveContainer" containerID="6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c"
Sep 16 18:54:57.255499 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:57.255465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc" event={"ID":"92bc46d0-34a8-44aa-97d8-dd13c100d25d","Type":"ContainerStarted","Data":"c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787"}
Sep 16 18:54:57.255922 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:54:57.255703    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:55:08.264972 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:08.264940    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 18:55:08.651047 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:08.651008    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c" podUID="4b254da9-78f9-457a-bf98-97a9cadfec99" containerName="main" containerID="cri-o://9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736" gracePeriod=45
Sep 16 18:55:08.681151 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:08.681120    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs" podUID="f7d7ce67-53f9-492f-9713-d319cd5ddeb6" containerName="main" containerID="cri-o://7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c" gracePeriod=45
Sep 16 18:55:08.960780 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:08.960755    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:55:08.963557 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:08.963535    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:55:09.019945 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.019916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-model-cache\") pod \"4b254da9-78f9-457a-bf98-97a9cadfec99\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") "
Sep 16 18:55:09.020132 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.019956    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-dshm\") pod \"4b254da9-78f9-457a-bf98-97a9cadfec99\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") "
Sep 16 18:55:09.020132 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.019981    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-tmp-dir\") pod \"4b254da9-78f9-457a-bf98-97a9cadfec99\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") "
Sep 16 18:55:09.020132 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.020010    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-home\") pod \"4b254da9-78f9-457a-bf98-97a9cadfec99\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") "
Sep 16 18:55:09.020132 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.020029    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b254da9-78f9-457a-bf98-97a9cadfec99-tls-certs\") pod \"4b254da9-78f9-457a-bf98-97a9cadfec99\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") "
Sep 16 18:55:09.020132 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.020050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmxbv\" (UniqueName: \"kubernetes.io/projected/4b254da9-78f9-457a-bf98-97a9cadfec99-kube-api-access-hmxbv\") pod \"4b254da9-78f9-457a-bf98-97a9cadfec99\" (UID: \"4b254da9-78f9-457a-bf98-97a9cadfec99\") "
Sep 16 18:55:09.020465 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.020256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b254da9-78f9-457a-bf98-97a9cadfec99" (UID: "4b254da9-78f9-457a-bf98-97a9cadfec99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:09.020465 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.020293    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b254da9-78f9-457a-bf98-97a9cadfec99" (UID: "4b254da9-78f9-457a-bf98-97a9cadfec99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:09.020465 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.020306    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-home" (OuterVolumeSpecName: "home") pod "4b254da9-78f9-457a-bf98-97a9cadfec99" (UID: "4b254da9-78f9-457a-bf98-97a9cadfec99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:09.020784 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.020761    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-model-cache\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.020868 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.020789    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-tmp-dir\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.020868 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.020802    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-home\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.022503 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.022472    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b254da9-78f9-457a-bf98-97a9cadfec99-kube-api-access-hmxbv" (OuterVolumeSpecName: "kube-api-access-hmxbv") pod "4b254da9-78f9-457a-bf98-97a9cadfec99" (UID: "4b254da9-78f9-457a-bf98-97a9cadfec99"). InnerVolumeSpecName "kube-api-access-hmxbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:09.022630 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.022522    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-dshm" (OuterVolumeSpecName: "dshm") pod "4b254da9-78f9-457a-bf98-97a9cadfec99" (UID: "4b254da9-78f9-457a-bf98-97a9cadfec99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:09.022630 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.022537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b254da9-78f9-457a-bf98-97a9cadfec99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b254da9-78f9-457a-bf98-97a9cadfec99" (UID: "4b254da9-78f9-457a-bf98-97a9cadfec99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:09.122053 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd2vh\" (UniqueName: \"kubernetes.io/projected/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-kube-api-access-fd2vh\") pod \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") "
Sep 16 18:55:09.122211 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tmp-dir\") pod \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") "
Sep 16 18:55:09.122211 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-home\") pod \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") "
Sep 16 18:55:09.122211 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-model-cache\") pod \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") "
Sep 16 18:55:09.122211 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122171    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-dshm\") pod \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") "
Sep 16 18:55:09.122473 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tls-certs\") pod \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\" (UID: \"f7d7ce67-53f9-492f-9713-d319cd5ddeb6\") "
Sep 16 18:55:09.122473 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122405    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7d7ce67-53f9-492f-9713-d319cd5ddeb6" (UID: "f7d7ce67-53f9-492f-9713-d319cd5ddeb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:09.122473 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-home" (OuterVolumeSpecName: "home") pod "f7d7ce67-53f9-492f-9713-d319cd5ddeb6" (UID: "f7d7ce67-53f9-492f-9713-d319cd5ddeb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:09.122473 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122461    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7d7ce67-53f9-492f-9713-d319cd5ddeb6" (UID: "f7d7ce67-53f9-492f-9713-d319cd5ddeb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:09.122672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122559    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tmp-dir\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.122672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122577    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-home\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.122672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122590    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-model-cache\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.122672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122606    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b254da9-78f9-457a-bf98-97a9cadfec99-dshm\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.122672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122618    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b254da9-78f9-457a-bf98-97a9cadfec99-tls-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.122672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.122631    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmxbv\" (UniqueName: \"kubernetes.io/projected/4b254da9-78f9-457a-bf98-97a9cadfec99-kube-api-access-hmxbv\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.124062 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.124039    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-kube-api-access-fd2vh" (OuterVolumeSpecName: "kube-api-access-fd2vh") pod "f7d7ce67-53f9-492f-9713-d319cd5ddeb6" (UID: "f7d7ce67-53f9-492f-9713-d319cd5ddeb6"). InnerVolumeSpecName "kube-api-access-fd2vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:09.124168 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.124149    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-dshm" (OuterVolumeSpecName: "dshm") pod "f7d7ce67-53f9-492f-9713-d319cd5ddeb6" (UID: "f7d7ce67-53f9-492f-9713-d319cd5ddeb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:09.124214 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.124154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7d7ce67-53f9-492f-9713-d319cd5ddeb6" (UID: "f7d7ce67-53f9-492f-9713-d319cd5ddeb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:09.223865 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.223789    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-dshm\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.223865 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.223814    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-tls-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.223865 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.223829    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd2vh\" (UniqueName: \"kubernetes.io/projected/f7d7ce67-53f9-492f-9713-d319cd5ddeb6-kube-api-access-fd2vh\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:09.331998 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.331958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs" event={"ID":"f7d7ce67-53f9-492f-9713-d319cd5ddeb6","Type":"ContainerDied","Data":"7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c"}
Sep 16 18:55:09.332534 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.332005    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"
Sep 16 18:55:09.332534 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.332017    2580 scope.go:117] "RemoveContainer" containerID="7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c"
Sep 16 18:55:09.332534 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.331960    2580 generic.go:356] "Generic (PLEG): container finished" podID="f7d7ce67-53f9-492f-9713-d319cd5ddeb6" containerID="7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c" exitCode=0
Sep 16 18:55:09.332534 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.332171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs" event={"ID":"f7d7ce67-53f9-492f-9713-d319cd5ddeb6","Type":"ContainerDied","Data":"ce6f5df940f08548c16fe2def2b3debbbc407c4739f584c00fd66d3f1da0fa1d"}
Sep 16 18:55:09.333822 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.333791    2580 generic.go:356] "Generic (PLEG): container finished" podID="4b254da9-78f9-457a-bf98-97a9cadfec99" containerID="9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736" exitCode=0
Sep 16 18:55:09.333975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.333854    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"
Sep 16 18:55:09.333975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.333856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c" event={"ID":"4b254da9-78f9-457a-bf98-97a9cadfec99","Type":"ContainerDied","Data":"9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736"}
Sep 16 18:55:09.333975 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.333953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c" event={"ID":"4b254da9-78f9-457a-bf98-97a9cadfec99","Type":"ContainerDied","Data":"6d014d6a7894e46d62e219db8e5aa7995ae94fcae8e418259090787bc864a7e9"}
Sep 16 18:55:09.342181 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.342036    2580 scope.go:117] "RemoveContainer" containerID="7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c"
Sep 16 18:55:09.342424 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:55:09.342382    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c\": container with ID starting with 7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c not found: ID does not exist" containerID="7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c"
Sep 16 18:55:09.342504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.342418    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c"} err="failed to get container status \"7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c\": rpc error: code = NotFound desc = could not find container \"7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c\": container with ID starting with 7c8d7743e93c56460a46c44708e8e661d2ed3a16df48a3131e31f51c9b0d4c2c not found: ID does not exist"
Sep 16 18:55:09.342504 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.342441    2580 scope.go:117] "RemoveContainer" containerID="9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736"
Sep 16 18:55:09.350568 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.350554    2580 scope.go:117] "RemoveContainer" containerID="9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736"
Sep 16 18:55:09.350813 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:55:09.350793    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736\": container with ID starting with 9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736 not found: ID does not exist" containerID="9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736"
Sep 16 18:55:09.350861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.350819    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736"} err="failed to get container status \"9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736\": rpc error: code = NotFound desc = could not find container \"9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736\": container with ID starting with 9fb58e9a27331208e213a0645eea810ba148d4df7086bd70c2d07caf81d2c736 not found: ID does not exist"
Sep 16 18:55:09.359807 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.359785    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"]
Sep 16 18:55:09.364879 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.364858    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/premium-w0-4dda98-kserve-8f54858fd-g9qcs"]
Sep 16 18:55:09.378349 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.378314    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"]
Sep 16 18:55:09.381937 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.381917    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4dda98/distinct-2-w0-4dda98-kserve-7dbf7b9dcd-b689c"]
Sep 16 18:55:09.783591 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.783562    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b254da9-78f9-457a-bf98-97a9cadfec99" path="/var/lib/kubelet/pods/4b254da9-78f9-457a-bf98-97a9cadfec99/volumes"
Sep 16 18:55:09.783960 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:09.783948    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d7ce67-53f9-492f-9713-d319cd5ddeb6" path="/var/lib/kubelet/pods/f7d7ce67-53f9-492f-9713-d319cd5ddeb6/volumes"
Sep 16 18:55:10.237098 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:10.237066    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"]
Sep 16 18:55:10.237292 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:10.237265    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp" podUID="8768ed75-5fea-4a32-84c1-22f0e6512de1" containerName="payload-pre-processing" containerID="cri-o://2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4" gracePeriod=30
Sep 16 18:55:10.432434 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:10.432406    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"]
Sep 16 18:55:10.435832 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:10.435808    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-41abff-fxjk8"]
Sep 16 18:55:10.491057 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:10.490992    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:55:10.636251 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:10.636224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdcwd\" (UniqueName: \"kubernetes.io/projected/8768ed75-5fea-4a32-84c1-22f0e6512de1-kube-api-access-fdcwd\") pod \"8768ed75-5fea-4a32-84c1-22f0e6512de1\" (UID: \"8768ed75-5fea-4a32-84c1-22f0e6512de1\") "
Sep 16 18:55:10.636425 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:10.636271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8768ed75-5fea-4a32-84c1-22f0e6512de1-plugins-config-volume\") pod \"8768ed75-5fea-4a32-84c1-22f0e6512de1\" (UID: \"8768ed75-5fea-4a32-84c1-22f0e6512de1\") "
Sep 16 18:55:10.636687 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:10.636662    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8768ed75-5fea-4a32-84c1-22f0e6512de1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8768ed75-5fea-4a32-84c1-22f0e6512de1" (UID: "8768ed75-5fea-4a32-84c1-22f0e6512de1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:10.638371 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:10.638300    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8768ed75-5fea-4a32-84c1-22f0e6512de1-kube-api-access-fdcwd" (OuterVolumeSpecName: "kube-api-access-fdcwd") pod "8768ed75-5fea-4a32-84c1-22f0e6512de1" (UID: "8768ed75-5fea-4a32-84c1-22f0e6512de1"). InnerVolumeSpecName "kube-api-access-fdcwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:10.737186 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:10.737157    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdcwd\" (UniqueName: \"kubernetes.io/projected/8768ed75-5fea-4a32-84c1-22f0e6512de1-kube-api-access-fdcwd\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:10.737186 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:10.737181    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8768ed75-5fea-4a32-84c1-22f0e6512de1-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:11.232221 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.232184    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"]
Sep 16 18:55:11.232487 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.232456    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx" podUID="fd91ebd5-4e35-4339-93c9-f4eedd8214db" containerName="payload-processing" containerID="cri-o://dc465c5dc1a70d7aba73844d463d9d47c71cfb292df9d08e847aeadae9b1634b" gracePeriod=30
Sep 16 18:55:11.238132 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.238104    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"]
Sep 16 18:55:11.238627 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.238324    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm" podUID="190586c8-8189-494e-b6c6-3d5491c15d56" containerName="payload-pre-processing" containerID="cri-o://ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa" gracePeriod=30
Sep 16 18:55:11.350510 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.350474    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd91ebd5-4e35-4339-93c9-f4eedd8214db" containerID="dc465c5dc1a70d7aba73844d463d9d47c71cfb292df9d08e847aeadae9b1634b" exitCode=0
Sep 16 18:55:11.350655 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.350580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx" event={"ID":"fd91ebd5-4e35-4339-93c9-f4eedd8214db","Type":"ContainerDied","Data":"dc465c5dc1a70d7aba73844d463d9d47c71cfb292df9d08e847aeadae9b1634b"}
Sep 16 18:55:11.352299 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.352275    2580 generic.go:356] "Generic (PLEG): container finished" podID="8768ed75-5fea-4a32-84c1-22f0e6512de1" containerID="2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4" exitCode=0
Sep 16 18:55:11.352461 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.352375    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"
Sep 16 18:55:11.352568 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.352374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp" event={"ID":"8768ed75-5fea-4a32-84c1-22f0e6512de1","Type":"ContainerDied","Data":"2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4"}
Sep 16 18:55:11.352685 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.352612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp" event={"ID":"8768ed75-5fea-4a32-84c1-22f0e6512de1","Type":"ContainerDied","Data":"3bc528cd3a19f201df2b630cbffc549052065bf503edb6b52ded2892fda6cea1"}
Sep 16 18:55:11.352685 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.352651    2580 scope.go:117] "RemoveContainer" containerID="2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4"
Sep 16 18:55:11.370997 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.370978    2580 scope.go:117] "RemoveContainer" containerID="2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4"
Sep 16 18:55:11.371290 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:55:11.371263    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4\": container with ID starting with 2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4 not found: ID does not exist" containerID="2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4"
Sep 16 18:55:11.371409 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.371294    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4"} err="failed to get container status \"2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4\": rpc error: code = NotFound desc = could not find container \"2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4\": container with ID starting with 2e0efdcd971b762241f7d87d57c45eba0de10ca6e9a7d382efb4a26b0b99c1d4 not found: ID does not exist"
Sep 16 18:55:11.382861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.382835    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"]
Sep 16 18:55:11.389375 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.389297    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-41abff-cf5f6cc8b-cw5jp"]
Sep 16 18:55:11.503422 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.503363    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:55:11.521245 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.521220    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:55:11.573636 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.573606    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"]
Sep 16 18:55:11.579016 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.578990    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4dda98-wc4j5"]
Sep 16 18:55:11.645842 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.645809    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd91ebd5-4e35-4339-93c9-f4eedd8214db-plugins-config-volume\") pod \"fd91ebd5-4e35-4339-93c9-f4eedd8214db\" (UID: \"fd91ebd5-4e35-4339-93c9-f4eedd8214db\") "
Sep 16 18:55:11.646023 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.645884    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4knl5\" (UniqueName: \"kubernetes.io/projected/fd91ebd5-4e35-4339-93c9-f4eedd8214db-kube-api-access-4knl5\") pod \"fd91ebd5-4e35-4339-93c9-f4eedd8214db\" (UID: \"fd91ebd5-4e35-4339-93c9-f4eedd8214db\") "
Sep 16 18:55:11.646023 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.645935    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190586c8-8189-494e-b6c6-3d5491c15d56-plugins-config-volume\") pod \"190586c8-8189-494e-b6c6-3d5491c15d56\" (UID: \"190586c8-8189-494e-b6c6-3d5491c15d56\") "
Sep 16 18:55:11.646023 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.645959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67vck\" (UniqueName: \"kubernetes.io/projected/190586c8-8189-494e-b6c6-3d5491c15d56-kube-api-access-67vck\") pod \"190586c8-8189-494e-b6c6-3d5491c15d56\" (UID: \"190586c8-8189-494e-b6c6-3d5491c15d56\") "
Sep 16 18:55:11.646255 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.646224    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd91ebd5-4e35-4339-93c9-f4eedd8214db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd91ebd5-4e35-4339-93c9-f4eedd8214db" (UID: "fd91ebd5-4e35-4339-93c9-f4eedd8214db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:11.646404 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.646263    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190586c8-8189-494e-b6c6-3d5491c15d56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "190586c8-8189-494e-b6c6-3d5491c15d56" (UID: "190586c8-8189-494e-b6c6-3d5491c15d56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:11.648205 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.648182    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190586c8-8189-494e-b6c6-3d5491c15d56-kube-api-access-67vck" (OuterVolumeSpecName: "kube-api-access-67vck") pod "190586c8-8189-494e-b6c6-3d5491c15d56" (UID: "190586c8-8189-494e-b6c6-3d5491c15d56"). InnerVolumeSpecName "kube-api-access-67vck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:11.648300 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.648214    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd91ebd5-4e35-4339-93c9-f4eedd8214db-kube-api-access-4knl5" (OuterVolumeSpecName: "kube-api-access-4knl5") pod "fd91ebd5-4e35-4339-93c9-f4eedd8214db" (UID: "fd91ebd5-4e35-4339-93c9-f4eedd8214db"). InnerVolumeSpecName "kube-api-access-4knl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:11.747180 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.747143    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd91ebd5-4e35-4339-93c9-f4eedd8214db-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:11.747180 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.747171    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4knl5\" (UniqueName: \"kubernetes.io/projected/fd91ebd5-4e35-4339-93c9-f4eedd8214db-kube-api-access-4knl5\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:11.747180 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.747181    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190586c8-8189-494e-b6c6-3d5491c15d56-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:11.747426 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.747190    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67vck\" (UniqueName: \"kubernetes.io/projected/190586c8-8189-494e-b6c6-3d5491c15d56-kube-api-access-67vck\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:11.783848 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.783820    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="209d00ec-8f7e-4e87-b70b-49d129a62ae6" path="/var/lib/kubelet/pods/209d00ec-8f7e-4e87-b70b-49d129a62ae6/volumes"
Sep 16 18:55:11.784148 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.784127    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42d69608-1f15-44d8-a2e8-cd0bc32f341b" path="/var/lib/kubelet/pods/42d69608-1f15-44d8-a2e8-cd0bc32f341b/volumes"
Sep 16 18:55:11.784474 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:11.784462    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8768ed75-5fea-4a32-84c1-22f0e6512de1" path="/var/lib/kubelet/pods/8768ed75-5fea-4a32-84c1-22f0e6512de1/volumes"
Sep 16 18:55:12.360027 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.359971    2580 generic.go:356] "Generic (PLEG): container finished" podID="190586c8-8189-494e-b6c6-3d5491c15d56" containerID="ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa" exitCode=0
Sep 16 18:55:12.360205 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.360034    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"
Sep 16 18:55:12.360205 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.360055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm" event={"ID":"190586c8-8189-494e-b6c6-3d5491c15d56","Type":"ContainerDied","Data":"ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa"}
Sep 16 18:55:12.360205 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.360141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm" event={"ID":"190586c8-8189-494e-b6c6-3d5491c15d56","Type":"ContainerDied","Data":"8300cafc4e93f3595cebaa990ab351c2c0c1f5f3a68d20c0754ad7bee8749b74"}
Sep 16 18:55:12.360205 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.360167    2580 scope.go:117] "RemoveContainer" containerID="ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa"
Sep 16 18:55:12.361669 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.361640    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"
Sep 16 18:55:12.361771 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.361644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx" event={"ID":"fd91ebd5-4e35-4339-93c9-f4eedd8214db","Type":"ContainerDied","Data":"4bf185f9967dfb206965e4e376f860427bcc5cc85381aef1b36eb0ea086165ab"}
Sep 16 18:55:12.369104 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.369090    2580 scope.go:117] "RemoveContainer" containerID="ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa"
Sep 16 18:55:12.369397 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:55:12.369376    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa\": container with ID starting with ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa not found: ID does not exist" containerID="ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa"
Sep 16 18:55:12.369457 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.369408    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa"} err="failed to get container status \"ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa\": rpc error: code = NotFound desc = could not find container \"ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa\": container with ID starting with ad108c31025b7448b64970ffc284610dd75d3a372f3e9d6a4bf6d5b471914baa not found: ID does not exist"
Sep 16 18:55:12.369457 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.369425    2580 scope.go:117] "RemoveContainer" containerID="dc465c5dc1a70d7aba73844d463d9d47c71cfb292df9d08e847aeadae9b1634b"
Sep 16 18:55:12.378152 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.378132    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"]
Sep 16 18:55:12.382978 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.382957    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4dda98-7449b5f466-s7wtm"]
Sep 16 18:55:12.394493 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.394470    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"]
Sep 16 18:55:12.401601 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:12.401573    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4dda98-86c4d4b76c-cxmlx"]
Sep 16 18:55:13.784604 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:13.784574    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="190586c8-8189-494e-b6c6-3d5491c15d56" path="/var/lib/kubelet/pods/190586c8-8189-494e-b6c6-3d5491c15d56/volumes"
Sep 16 18:55:13.784990 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:13.784975    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd91ebd5-4e35-4339-93c9-f4eedd8214db" path="/var/lib/kubelet/pods/fd91ebd5-4e35-4339-93c9-f4eedd8214db/volumes"
Sep 16 18:55:18.283544 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283510    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"]
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283906    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b254da9-78f9-457a-bf98-97a9cadfec99" containerName="main"
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283919    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b254da9-78f9-457a-bf98-97a9cadfec99" containerName="main"
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283925    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209d00ec-8f7e-4e87-b70b-49d129a62ae6" containerName="revoke-keys"
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283931    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="209d00ec-8f7e-4e87-b70b-49d129a62ae6" containerName="revoke-keys"
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283943    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d7ce67-53f9-492f-9713-d319cd5ddeb6" containerName="main"
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283948    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d7ce67-53f9-492f-9713-d319cd5ddeb6" containerName="main"
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283955    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42d69608-1f15-44d8-a2e8-cd0bc32f341b" containerName="revoke-keys"
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283963    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d69608-1f15-44d8-a2e8-cd0bc32f341b" containerName="revoke-keys"
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283973    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8768ed75-5fea-4a32-84c1-22f0e6512de1" containerName="payload-pre-processing"
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283978    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8768ed75-5fea-4a32-84c1-22f0e6512de1" containerName="payload-pre-processing"
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283987    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190586c8-8189-494e-b6c6-3d5491c15d56" containerName="payload-pre-processing"
Sep 16 18:55:18.283991 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.283993    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="190586c8-8189-494e-b6c6-3d5491c15d56" containerName="payload-pre-processing"
Sep 16 18:55:18.284437 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.284001    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd91ebd5-4e35-4339-93c9-f4eedd8214db" containerName="payload-processing"
Sep 16 18:55:18.284437 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.284006    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd91ebd5-4e35-4339-93c9-f4eedd8214db" containerName="payload-processing"
Sep 16 18:55:18.284437 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.284069    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="42d69608-1f15-44d8-a2e8-cd0bc32f341b" containerName="revoke-keys"
Sep 16 18:55:18.284437 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.284078    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b254da9-78f9-457a-bf98-97a9cadfec99" containerName="main"
Sep 16 18:55:18.284437 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.284085    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d7ce67-53f9-492f-9713-d319cd5ddeb6" containerName="main"
Sep 16 18:55:18.284437 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.284091    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="190586c8-8189-494e-b6c6-3d5491c15d56" containerName="payload-pre-processing"
Sep 16 18:55:18.284437 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.284096    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8768ed75-5fea-4a32-84c1-22f0e6512de1" containerName="payload-pre-processing"
Sep 16 18:55:18.284437 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.284102    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="209d00ec-8f7e-4e87-b70b-49d129a62ae6" containerName="revoke-keys"
Sep 16 18:55:18.284437 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.284108    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd91ebd5-4e35-4339-93c9-f4eedd8214db" containerName="payload-processing"
Sep 16 18:55:18.288880 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.288864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"
Sep 16 18:55:18.291140 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.291119    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:55:18.291260 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.291119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:55:18.295364 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.295321    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"]
Sep 16 18:55:18.412150 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.412115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d8779a-b749-49ef-bbc7-e2b79ec9859d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4\" (UID: \"42d8779a-b749-49ef-bbc7-e2b79ec9859d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"
Sep 16 18:55:18.513587 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.513549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d8779a-b749-49ef-bbc7-e2b79ec9859d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4\" (UID: \"42d8779a-b749-49ef-bbc7-e2b79ec9859d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"
Sep 16 18:55:18.514131 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.514112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d8779a-b749-49ef-bbc7-e2b79ec9859d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4\" (UID: \"42d8779a-b749-49ef-bbc7-e2b79ec9859d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"
Sep 16 18:55:18.600714 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.600631    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"
Sep 16 18:55:18.873044 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:18.873017    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"]
Sep 16 18:55:18.875245 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:55:18.875208    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d8779a_b749_49ef_bbc7_e2b79ec9859d.slice/crio-2d7929bf4466158bc78008b1f81948d38fa78f7593d7f0cd6a62825f5cf8a2d8 WatchSource:0}: Error finding container 2d7929bf4466158bc78008b1f81948d38fa78f7593d7f0cd6a62825f5cf8a2d8: Status 404 returned error can't find the container with id 2d7929bf4466158bc78008b1f81948d38fa78f7593d7f0cd6a62825f5cf8a2d8
Sep 16 18:55:19.402577 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:19.402543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4" event={"ID":"42d8779a-b749-49ef-bbc7-e2b79ec9859d","Type":"ContainerStarted","Data":"9f2910b71f379b7c317b35c90d1b6fbd2a002ea08fdf87684a146ab882d80446"}
Sep 16 18:55:19.402953 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:19.402596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4" event={"ID":"42d8779a-b749-49ef-bbc7-e2b79ec9859d","Type":"ContainerStarted","Data":"2d7929bf4466158bc78008b1f81948d38fa78f7593d7f0cd6a62825f5cf8a2d8"}
Sep 16 18:55:19.421025 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:19.420981    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4" podStartSLOduration=1.420965097 podStartE2EDuration="1.420965097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:19.419982947 +0000 UTC m=+1662.129222341" watchObservedRunningTime="2026-09-16 18:55:19.420965097 +0000 UTC m=+1662.130204489"
Sep 16 18:55:20.409443 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:20.409408    2580 generic.go:356] "Generic (PLEG): container finished" podID="42d8779a-b749-49ef-bbc7-e2b79ec9859d" containerID="9f2910b71f379b7c317b35c90d1b6fbd2a002ea08fdf87684a146ab882d80446" exitCode=0
Sep 16 18:55:20.409854 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:20.409500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4" event={"ID":"42d8779a-b749-49ef-bbc7-e2b79ec9859d","Type":"ContainerDied","Data":"9f2910b71f379b7c317b35c90d1b6fbd2a002ea08fdf87684a146ab882d80446"}
Sep 16 18:55:21.539815 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:21.539778    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"
Sep 16 18:55:21.645052 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:21.645018    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d8779a-b749-49ef-bbc7-e2b79ec9859d-maas-api-service-ca\") pod \"42d8779a-b749-49ef-bbc7-e2b79ec9859d\" (UID: \"42d8779a-b749-49ef-bbc7-e2b79ec9859d\") "
Sep 16 18:55:21.645433 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:21.645398    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d8779a-b749-49ef-bbc7-e2b79ec9859d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42d8779a-b749-49ef-bbc7-e2b79ec9859d" (UID: "42d8779a-b749-49ef-bbc7-e2b79ec9859d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:21.746174 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:21.746096    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d8779a-b749-49ef-bbc7-e2b79ec9859d-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:22.422340 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:22.422310    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"
Sep 16 18:55:22.422515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:22.422312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4" event={"ID":"42d8779a-b749-49ef-bbc7-e2b79ec9859d","Type":"ContainerDied","Data":"2d7929bf4466158bc78008b1f81948d38fa78f7593d7f0cd6a62825f5cf8a2d8"}
Sep 16 18:55:22.422515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:22.422457    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d7929bf4466158bc78008b1f81948d38fa78f7593d7f0cd6a62825f5cf8a2d8"
Sep 16 18:55:29.132252 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:29.132221    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:55:33.696103 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:33.696071    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"]
Sep 16 18:55:33.696513 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:33.696288    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr" podUID="80e45844-e9ee-4733-94b8-a91ad94a643d" containerName="payload-processing" containerID="cri-o://e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c" gracePeriod=30
Sep 16 18:55:33.706862 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:33.706831    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"]
Sep 16 18:55:33.707111 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:33.707084    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6" podUID="eb8f4b0f-35e5-4346-823d-90b1c71e16ac" containerName="payload-pre-processing" containerID="cri-o://4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884" gracePeriod=30
Sep 16 18:55:33.966696 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:33.966669    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"]
Sep 16 18:55:33.972499 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:33.972479    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ee98a5-5xjk4"]
Sep 16 18:55:33.981148 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:33.981126    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:55:33.983967 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:33.983951    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:55:34.070224 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.070193    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e45844-e9ee-4733-94b8-a91ad94a643d-plugins-config-volume\") pod \"80e45844-e9ee-4733-94b8-a91ad94a643d\" (UID: \"80e45844-e9ee-4733-94b8-a91ad94a643d\") "
Sep 16 18:55:34.070452 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.070231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-plugins-config-volume\") pod \"eb8f4b0f-35e5-4346-823d-90b1c71e16ac\" (UID: \"eb8f4b0f-35e5-4346-823d-90b1c71e16ac\") "
Sep 16 18:55:34.070452 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.070414    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2swgw\" (UniqueName: \"kubernetes.io/projected/80e45844-e9ee-4733-94b8-a91ad94a643d-kube-api-access-2swgw\") pod \"80e45844-e9ee-4733-94b8-a91ad94a643d\" (UID: \"80e45844-e9ee-4733-94b8-a91ad94a643d\") "
Sep 16 18:55:34.070452 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.070437    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4w9s\" (UniqueName: \"kubernetes.io/projected/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-kube-api-access-x4w9s\") pod \"eb8f4b0f-35e5-4346-823d-90b1c71e16ac\" (UID: \"eb8f4b0f-35e5-4346-823d-90b1c71e16ac\") "
Sep 16 18:55:34.070696 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.070672    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb8f4b0f-35e5-4346-823d-90b1c71e16ac" (UID: "eb8f4b0f-35e5-4346-823d-90b1c71e16ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:34.070761 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.070681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e45844-e9ee-4733-94b8-a91ad94a643d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80e45844-e9ee-4733-94b8-a91ad94a643d" (UID: "80e45844-e9ee-4733-94b8-a91ad94a643d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:34.070832 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.070785    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:34.070832 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.070802    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e45844-e9ee-4733-94b8-a91ad94a643d-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:34.072633 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.072609    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-kube-api-access-x4w9s" (OuterVolumeSpecName: "kube-api-access-x4w9s") pod "eb8f4b0f-35e5-4346-823d-90b1c71e16ac" (UID: "eb8f4b0f-35e5-4346-823d-90b1c71e16ac"). InnerVolumeSpecName "kube-api-access-x4w9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:34.072810 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.072789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e45844-e9ee-4733-94b8-a91ad94a643d-kube-api-access-2swgw" (OuterVolumeSpecName: "kube-api-access-2swgw") pod "80e45844-e9ee-4733-94b8-a91ad94a643d" (UID: "80e45844-e9ee-4733-94b8-a91ad94a643d"). InnerVolumeSpecName "kube-api-access-2swgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:34.171602 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.171564    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2swgw\" (UniqueName: \"kubernetes.io/projected/80e45844-e9ee-4733-94b8-a91ad94a643d-kube-api-access-2swgw\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:34.171602 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.171593    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4w9s\" (UniqueName: \"kubernetes.io/projected/eb8f4b0f-35e5-4346-823d-90b1c71e16ac-kube-api-access-x4w9s\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:34.488372 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.488309    2580 generic.go:356] "Generic (PLEG): container finished" podID="80e45844-e9ee-4733-94b8-a91ad94a643d" containerID="e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c" exitCode=0
Sep 16 18:55:34.488372 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.488371    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"
Sep 16 18:55:34.488606 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.488365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr" event={"ID":"80e45844-e9ee-4733-94b8-a91ad94a643d","Type":"ContainerDied","Data":"e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c"}
Sep 16 18:55:34.488606 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.488492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr" event={"ID":"80e45844-e9ee-4733-94b8-a91ad94a643d","Type":"ContainerDied","Data":"3e1403d9b47748805832b9492603ce051ef4336b7675462b6e8fd1fb688e27e0"}
Sep 16 18:55:34.488606 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.488515    2580 scope.go:117] "RemoveContainer" containerID="e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c"
Sep 16 18:55:34.489908 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.489875    2580 generic.go:356] "Generic (PLEG): container finished" podID="eb8f4b0f-35e5-4346-823d-90b1c71e16ac" containerID="4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884" exitCode=0
Sep 16 18:55:34.490020 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.489906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6" event={"ID":"eb8f4b0f-35e5-4346-823d-90b1c71e16ac","Type":"ContainerDied","Data":"4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884"}
Sep 16 18:55:34.490020 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.489961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6" event={"ID":"eb8f4b0f-35e5-4346-823d-90b1c71e16ac","Type":"ContainerDied","Data":"33ebf21b00ceb2e1eb9a08e6ca9832604ab08c16420be9a00b6aeac4cde7166e"}
Sep 16 18:55:34.490020 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.489917    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"
Sep 16 18:55:34.498103 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.498082    2580 scope.go:117] "RemoveContainer" containerID="e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c"
Sep 16 18:55:34.498417 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:55:34.498398    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c\": container with ID starting with e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c not found: ID does not exist" containerID="e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c"
Sep 16 18:55:34.498515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.498430    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c"} err="failed to get container status \"e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c\": rpc error: code = NotFound desc = could not find container \"e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c\": container with ID starting with e15751419d231b10ff37990caa810cd688b334446c146c25536dfdb3fa3ae90c not found: ID does not exist"
Sep 16 18:55:34.498515 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.498451    2580 scope.go:117] "RemoveContainer" containerID="4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884"
Sep 16 18:55:34.506799 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.506763    2580 scope.go:117] "RemoveContainer" containerID="4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884"
Sep 16 18:55:34.507036 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:55:34.507018    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884\": container with ID starting with 4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884 not found: ID does not exist" containerID="4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884"
Sep 16 18:55:34.507091 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.507043    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884"} err="failed to get container status \"4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884\": rpc error: code = NotFound desc = could not find container \"4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884\": container with ID starting with 4dc522ece26e674c5fbbd4cacc32d4f4979474e23c71a10946fe587e5c27d884 not found: ID does not exist"
Sep 16 18:55:34.527635 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.527602    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"]
Sep 16 18:55:34.530730 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.530711    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ee98a5-6c8d57fcf5-wh9sr"]
Sep 16 18:55:34.552647 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.552626    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"]
Sep 16 18:55:34.571317 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:34.571292    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ee98a5-5859df9cf4-772p6"]
Sep 16 18:55:35.783994 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:35.783950    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42d8779a-b749-49ef-bbc7-e2b79ec9859d" path="/var/lib/kubelet/pods/42d8779a-b749-49ef-bbc7-e2b79ec9859d/volumes"
Sep 16 18:55:35.784550 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:35.784528    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e45844-e9ee-4733-94b8-a91ad94a643d" path="/var/lib/kubelet/pods/80e45844-e9ee-4733-94b8-a91ad94a643d/volumes"
Sep 16 18:55:35.784934 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:35.784915    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb8f4b0f-35e5-4346-823d-90b1c71e16ac" path="/var/lib/kubelet/pods/eb8f4b0f-35e5-4346-823d-90b1c71e16ac/volumes"
Sep 16 18:55:42.945129 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.945094    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"]
Sep 16 18:55:42.945687 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.945670    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42d8779a-b749-49ef-bbc7-e2b79ec9859d" containerName="revoke-keys"
Sep 16 18:55:42.945731 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.945690    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d8779a-b749-49ef-bbc7-e2b79ec9859d" containerName="revoke-keys"
Sep 16 18:55:42.945731 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.945706    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8f4b0f-35e5-4346-823d-90b1c71e16ac" containerName="payload-pre-processing"
Sep 16 18:55:42.945731 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.945715    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8f4b0f-35e5-4346-823d-90b1c71e16ac" containerName="payload-pre-processing"
Sep 16 18:55:42.945822 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.945734    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e45844-e9ee-4733-94b8-a91ad94a643d" containerName="payload-processing"
Sep 16 18:55:42.945822 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.945742    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e45844-e9ee-4733-94b8-a91ad94a643d" containerName="payload-processing"
Sep 16 18:55:42.945889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.945843    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e45844-e9ee-4733-94b8-a91ad94a643d" containerName="payload-processing"
Sep 16 18:55:42.945889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.945856    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="42d8779a-b749-49ef-bbc7-e2b79ec9859d" containerName="revoke-keys"
Sep 16 18:55:42.945889 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.945865    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8f4b0f-35e5-4346-823d-90b1c71e16ac" containerName="payload-pre-processing"
Sep 16 18:55:42.948878 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.948860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"
Sep 16 18:55:42.951378 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.951350    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:55:42.951510 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.951399    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:55:42.959042 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:42.959009    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"]
Sep 16 18:55:43.052248 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:43.052216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ede01f-e91c-4caf-a4cb-9270680b1ebe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw\" (UID: \"62ede01f-e91c-4caf-a4cb-9270680b1ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"
Sep 16 18:55:43.153626 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:43.153590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ede01f-e91c-4caf-a4cb-9270680b1ebe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw\" (UID: \"62ede01f-e91c-4caf-a4cb-9270680b1ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"
Sep 16 18:55:43.154220 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:43.154201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ede01f-e91c-4caf-a4cb-9270680b1ebe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw\" (UID: \"62ede01f-e91c-4caf-a4cb-9270680b1ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"
Sep 16 18:55:43.263067 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:43.262967    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"
Sep 16 18:55:43.534860 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:43.534836    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"]
Sep 16 18:55:43.536993 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:55:43.536949    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ede01f_e91c_4caf_a4cb_9270680b1ebe.slice/crio-d4ff52bcaab902698b40b57c99def6e333e2197dbd93ce6d6d91f3c3f0458555 WatchSource:0}: Error finding container d4ff52bcaab902698b40b57c99def6e333e2197dbd93ce6d6d91f3c3f0458555: Status 404 returned error can't find the container with id d4ff52bcaab902698b40b57c99def6e333e2197dbd93ce6d6d91f3c3f0458555
Sep 16 18:55:43.546723 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:43.546691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw" event={"ID":"62ede01f-e91c-4caf-a4cb-9270680b1ebe","Type":"ContainerStarted","Data":"d4ff52bcaab902698b40b57c99def6e333e2197dbd93ce6d6d91f3c3f0458555"}
Sep 16 18:55:44.553645 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:44.553610    2580 generic.go:356] "Generic (PLEG): container finished" podID="62ede01f-e91c-4caf-a4cb-9270680b1ebe" containerID="7199a3b562281c1b39d1cc5c8416b083c08c39a3ea6dd923bce559c8255f759e" exitCode=0
Sep 16 18:55:44.554019 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:44.553694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw" event={"ID":"62ede01f-e91c-4caf-a4cb-9270680b1ebe","Type":"ContainerDied","Data":"7199a3b562281c1b39d1cc5c8416b083c08c39a3ea6dd923bce559c8255f759e"}
Sep 16 18:55:45.687685 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:45.687658    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"
Sep 16 18:55:45.781439 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:45.781407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ede01f-e91c-4caf-a4cb-9270680b1ebe-maas-api-service-ca\") pod \"62ede01f-e91c-4caf-a4cb-9270680b1ebe\" (UID: \"62ede01f-e91c-4caf-a4cb-9270680b1ebe\") "
Sep 16 18:55:45.781763 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:45.781740    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ede01f-e91c-4caf-a4cb-9270680b1ebe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62ede01f-e91c-4caf-a4cb-9270680b1ebe" (UID: "62ede01f-e91c-4caf-a4cb-9270680b1ebe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:45.882394 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:45.882279    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ede01f-e91c-4caf-a4cb-9270680b1ebe-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:46.565291 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:46.565264    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"
Sep 16 18:55:46.565482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:46.565311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw" event={"ID":"62ede01f-e91c-4caf-a4cb-9270680b1ebe","Type":"ContainerDied","Data":"d4ff52bcaab902698b40b57c99def6e333e2197dbd93ce6d6d91f3c3f0458555"}
Sep 16 18:55:46.565482 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:46.565377    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4ff52bcaab902698b40b57c99def6e333e2197dbd93ce6d6d91f3c3f0458555"
Sep 16 18:55:53.905761 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:53.905729    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"]
Sep 16 18:55:53.906234 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:53.905997    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6" podUID="96644388-feb0-4638-9fdb-4df53d0d523a" containerName="payload-pre-processing" containerID="cri-o://748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa" gracePeriod=30
Sep 16 18:55:54.080275 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.080245    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"]
Sep 16 18:55:54.083306 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.083279    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8737d2-cqfbw"]
Sep 16 18:55:54.167857 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.167779    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:55:54.262302 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.262271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v95t\" (UniqueName: \"kubernetes.io/projected/96644388-feb0-4638-9fdb-4df53d0d523a-kube-api-access-5v95t\") pod \"96644388-feb0-4638-9fdb-4df53d0d523a\" (UID: \"96644388-feb0-4638-9fdb-4df53d0d523a\") "
Sep 16 18:55:54.262503 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.262423    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96644388-feb0-4638-9fdb-4df53d0d523a-plugins-config-volume\") pod \"96644388-feb0-4638-9fdb-4df53d0d523a\" (UID: \"96644388-feb0-4638-9fdb-4df53d0d523a\") "
Sep 16 18:55:54.262846 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.262818    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96644388-feb0-4638-9fdb-4df53d0d523a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96644388-feb0-4638-9fdb-4df53d0d523a" (UID: "96644388-feb0-4638-9fdb-4df53d0d523a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:54.264560 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.264529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96644388-feb0-4638-9fdb-4df53d0d523a-kube-api-access-5v95t" (OuterVolumeSpecName: "kube-api-access-5v95t") pod "96644388-feb0-4638-9fdb-4df53d0d523a" (UID: "96644388-feb0-4638-9fdb-4df53d0d523a"). InnerVolumeSpecName "kube-api-access-5v95t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:54.364005 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.363969    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v95t\" (UniqueName: \"kubernetes.io/projected/96644388-feb0-4638-9fdb-4df53d0d523a-kube-api-access-5v95t\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:54.364005 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.363999    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96644388-feb0-4638-9fdb-4df53d0d523a-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:55:54.608745 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.608707    2580 generic.go:356] "Generic (PLEG): container finished" podID="96644388-feb0-4638-9fdb-4df53d0d523a" containerID="748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa" exitCode=0
Sep 16 18:55:54.608970 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.608750    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"
Sep 16 18:55:54.608970 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.608794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6" event={"ID":"96644388-feb0-4638-9fdb-4df53d0d523a","Type":"ContainerDied","Data":"748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa"}
Sep 16 18:55:54.608970 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.608857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6" event={"ID":"96644388-feb0-4638-9fdb-4df53d0d523a","Type":"ContainerDied","Data":"72ce99f8d89c7f61e825f5e79706eb1e1f27a66a82030ebd0a06b94ad4f27dca"}
Sep 16 18:55:54.608970 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.608885    2580 scope.go:117] "RemoveContainer" containerID="748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa"
Sep 16 18:55:54.618347 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.618306    2580 scope.go:117] "RemoveContainer" containerID="748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa"
Sep 16 18:55:54.618707 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:55:54.618676    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa\": container with ID starting with 748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa not found: ID does not exist" containerID="748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa"
Sep 16 18:55:54.618807 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.618717    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa"} err="failed to get container status \"748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa\": rpc error: code = NotFound desc = could not find container \"748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa\": container with ID starting with 748e06245023212af8c93efdd7a64da89c23249b007f09f7848e509a14592cfa not found: ID does not exist"
Sep 16 18:55:54.652827 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.652796    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"]
Sep 16 18:55:54.656198 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:54.656173    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8737d2-74fb4565f8-j6kr6"]
Sep 16 18:55:55.783219 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:55.783179    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ede01f-e91c-4caf-a4cb-9270680b1ebe" path="/var/lib/kubelet/pods/62ede01f-e91c-4caf-a4cb-9270680b1ebe/volumes"
Sep 16 18:55:55.783747 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:55:55.783722    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96644388-feb0-4638-9fdb-4df53d0d523a" path="/var/lib/kubelet/pods/96644388-feb0-4638-9fdb-4df53d0d523a/volumes"
Sep 16 18:56:00.925891 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:00.925806    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:56:01.745111 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:01.745080    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"]
Sep 16 18:56:01.745387 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:01.745316    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy" containerID="cri-o://70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61" gracePeriod=30
Sep 16 18:56:06.991821 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:06.991791    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:56:07.090254 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8wtd\" (UniqueName: \"kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-kube-api-access-h8wtd\") pod \"cb167c06-6fa4-4437-9994-709b682d6634\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") "
Sep 16 18:56:07.090486 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090309    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-certs\") pod \"cb167c06-6fa4-4437-9994-709b682d6634\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") "
Sep 16 18:56:07.090486 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090404    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb167c06-6fa4-4437-9994-709b682d6634-istio-podinfo\") pod \"cb167c06-6fa4-4437-9994-709b682d6634\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") "
Sep 16 18:56:07.090486 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb167c06-6fa4-4437-9994-709b682d6634-istiod-ca-cert\") pod \"cb167c06-6fa4-4437-9994-709b682d6634\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") "
Sep 16 18:56:07.090672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-istio-token\") pod \"cb167c06-6fa4-4437-9994-709b682d6634\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") "
Sep 16 18:56:07.090672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090537    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-data\") pod \"cb167c06-6fa4-4437-9994-709b682d6634\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") "
Sep 16 18:56:07.090672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090571    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-socket\") pod \"cb167c06-6fa4-4437-9994-709b682d6634\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") "
Sep 16 18:56:07.090672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090596    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb167c06-6fa4-4437-9994-709b682d6634" (UID: "cb167c06-6fa4-4437-9994-709b682d6634"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:07.090672 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090604    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-credential-socket\") pod \"cb167c06-6fa4-4437-9994-709b682d6634\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") "
Sep 16 18:56:07.090903 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090679    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-envoy\") pod \"cb167c06-6fa4-4437-9994-709b682d6634\" (UID: \"cb167c06-6fa4-4437-9994-709b682d6634\") "
Sep 16 18:56:07.090903 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb167c06-6fa4-4437-9994-709b682d6634" (UID: "cb167c06-6fa4-4437-9994-709b682d6634"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:07.090903 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090859    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb167c06-6fa4-4437-9994-709b682d6634" (UID: "cb167c06-6fa4-4437-9994-709b682d6634"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:07.090903 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.090868    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb167c06-6fa4-4437-9994-709b682d6634-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb167c06-6fa4-4437-9994-709b682d6634" (UID: "cb167c06-6fa4-4437-9994-709b682d6634"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:07.091083 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.091015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb167c06-6fa4-4437-9994-709b682d6634" (UID: "cb167c06-6fa4-4437-9994-709b682d6634"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:07.091083 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.091071    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:07.091174 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.091090    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb167c06-6fa4-4437-9994-709b682d6634-istiod-ca-cert\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:07.091174 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.091103    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-data\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:07.091174 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.091116    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-credential-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:07.092519 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.092492    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb167c06-6fa4-4437-9994-709b682d6634-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb167c06-6fa4-4437-9994-709b682d6634" (UID: "cb167c06-6fa4-4437-9994-709b682d6634"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:56:07.092742 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.092726    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb167c06-6fa4-4437-9994-709b682d6634" (UID: "cb167c06-6fa4-4437-9994-709b682d6634"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:07.092911 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.092895    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-kube-api-access-h8wtd" (OuterVolumeSpecName: "kube-api-access-h8wtd") pod "cb167c06-6fa4-4437-9994-709b682d6634" (UID: "cb167c06-6fa4-4437-9994-709b682d6634"). InnerVolumeSpecName "kube-api-access-h8wtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:07.092979 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.092908    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb167c06-6fa4-4437-9994-709b682d6634" (UID: "cb167c06-6fa4-4437-9994-709b682d6634"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:07.191578 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.191531    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb167c06-6fa4-4437-9994-709b682d6634-istio-podinfo\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:07.191578 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.191574    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-istio-token\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:07.191578 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.191584    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-workload-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:07.191578 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.191593    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb167c06-6fa4-4437-9994-709b682d6634-istio-envoy\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:07.191866 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.191602    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8wtd\" (UniqueName: \"kubernetes.io/projected/cb167c06-6fa4-4437-9994-709b682d6634-kube-api-access-h8wtd\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:07.686680 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.686648    2580 generic.go:356] "Generic (PLEG): container finished" podID="cb167c06-6fa4-4437-9994-709b682d6634" containerID="70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61" exitCode=0
Sep 16 18:56:07.686861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.686705    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"
Sep 16 18:56:07.686861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.686735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" event={"ID":"cb167c06-6fa4-4437-9994-709b682d6634","Type":"ContainerDied","Data":"70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61"}
Sep 16 18:56:07.686861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.686791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt" event={"ID":"cb167c06-6fa4-4437-9994-709b682d6634","Type":"ContainerDied","Data":"709b6d0c377a2729f681089d28a26a97e619144f7d896af39ff08a2d5d0f8a28"}
Sep 16 18:56:07.686861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.686813    2580 scope.go:117] "RemoveContainer" containerID="70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61"
Sep 16 18:56:07.696675 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.696658    2580 scope.go:117] "RemoveContainer" containerID="55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0"
Sep 16 18:56:07.704270 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.704254    2580 scope.go:117] "RemoveContainer" containerID="70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61"
Sep 16 18:56:07.704544 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:56:07.704528    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61\": container with ID starting with 70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61 not found: ID does not exist" containerID="70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61"
Sep 16 18:56:07.704601 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.704551    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61"} err="failed to get container status \"70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61\": rpc error: code = NotFound desc = could not find container \"70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61\": container with ID starting with 70371cd53b41a0960ead5a01bd63dadaae9b2f86b06778f6faa5935242f64d61 not found: ID does not exist"
Sep 16 18:56:07.704601 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.704565    2580 scope.go:117] "RemoveContainer" containerID="55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0"
Sep 16 18:56:07.704821 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:56:07.704804    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0\": container with ID starting with 55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0 not found: ID does not exist" containerID="55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0"
Sep 16 18:56:07.704858 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.704829    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0"} err="failed to get container status \"55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0\": rpc error: code = NotFound desc = could not find container \"55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0\": container with ID starting with 55c35823021af3039ca89bf4efb6222a9fa80702a9bc3b2613cdb2c79db150f0 not found: ID does not exist"
Sep 16 18:56:07.713613 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.713591    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"]
Sep 16 18:56:07.717928 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.717909    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8737d2-openshift-default-6fd4f95d-vdkgt"]
Sep 16 18:56:07.783926 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:07.783895    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb167c06-6fa4-4437-9994-709b682d6634" path="/var/lib/kubelet/pods/cb167c06-6fa4-4437-9994-709b682d6634/volumes"
Sep 16 18:56:22.127247 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:22.127212    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:56:27.476930 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.476891    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"]
Sep 16 18:56:27.477476 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.477462    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ede01f-e91c-4caf-a4cb-9270680b1ebe" containerName="revoke-keys"
Sep 16 18:56:27.477558 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.477482    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ede01f-e91c-4caf-a4cb-9270680b1ebe" containerName="revoke-keys"
Sep 16 18:56:27.477558 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.477495    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy"
Sep 16 18:56:27.477558 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.477505    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy"
Sep 16 18:56:27.477558 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.477530    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy"
Sep 16 18:56:27.477558 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.477540    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy"
Sep 16 18:56:27.477558 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.477554    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96644388-feb0-4638-9fdb-4df53d0d523a" containerName="payload-pre-processing"
Sep 16 18:56:27.477879 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.477562    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="96644388-feb0-4638-9fdb-4df53d0d523a" containerName="payload-pre-processing"
Sep 16 18:56:27.477879 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.477663    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="96644388-feb0-4638-9fdb-4df53d0d523a" containerName="payload-pre-processing"
Sep 16 18:56:27.477879 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.477674    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy"
Sep 16 18:56:27.477879 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.477686    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ede01f-e91c-4caf-a4cb-9270680b1ebe" containerName="revoke-keys"
Sep 16 18:56:27.480842 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.480820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"
Sep 16 18:56:27.483436 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.483414    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:56:27.483549 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.483463    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 18:56:27.488725 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.488699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"]
Sep 16 18:56:27.582219 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.582182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7753d-250d-4b6c-a204-3ac679d8c0d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-662432-9g58m\" (UID: \"8ba7753d-250d-4b6c-a204-3ac679d8c0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"
Sep 16 18:56:27.682993 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.682959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7753d-250d-4b6c-a204-3ac679d8c0d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-662432-9g58m\" (UID: \"8ba7753d-250d-4b6c-a204-3ac679d8c0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"
Sep 16 18:56:27.683788 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.683752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7753d-250d-4b6c-a204-3ac679d8c0d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-662432-9g58m\" (UID: \"8ba7753d-250d-4b6c-a204-3ac679d8c0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"
Sep 16 18:56:27.793038 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:27.793005    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"
Sep 16 18:56:28.066023 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:28.065993    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"]
Sep 16 18:56:28.067662 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:56:28.067620    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba7753d_250d_4b6c_a204_3ac679d8c0d5.slice/crio-8533f0e5bb394b79eb03aadebde94437cf4465b0d497f55dd823c96600ecd689 WatchSource:0}: Error finding container 8533f0e5bb394b79eb03aadebde94437cf4465b0d497f55dd823c96600ecd689: Status 404 returned error can't find the container with id 8533f0e5bb394b79eb03aadebde94437cf4465b0d497f55dd823c96600ecd689
Sep 16 18:56:28.797011 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:28.796978    2580 generic.go:356] "Generic (PLEG): container finished" podID="8ba7753d-250d-4b6c-a204-3ac679d8c0d5" containerID="1d482e5f0ef619a2f46b27f57b9453295ab72545b7ad47fae4859f7809933a25" exitCode=0
Sep 16 18:56:28.797481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:28.797063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m" event={"ID":"8ba7753d-250d-4b6c-a204-3ac679d8c0d5","Type":"ContainerDied","Data":"1d482e5f0ef619a2f46b27f57b9453295ab72545b7ad47fae4859f7809933a25"}
Sep 16 18:56:28.797481 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:28.797113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m" event={"ID":"8ba7753d-250d-4b6c-a204-3ac679d8c0d5","Type":"ContainerStarted","Data":"8533f0e5bb394b79eb03aadebde94437cf4465b0d497f55dd823c96600ecd689"}
Sep 16 18:56:29.931427 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:29.931403    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"
Sep 16 18:56:30.002778 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:30.002748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7753d-250d-4b6c-a204-3ac679d8c0d5-maas-api-service-ca\") pod \"8ba7753d-250d-4b6c-a204-3ac679d8c0d5\" (UID: \"8ba7753d-250d-4b6c-a204-3ac679d8c0d5\") "
Sep 16 18:56:30.003106 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:30.003084    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba7753d-250d-4b6c-a204-3ac679d8c0d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ba7753d-250d-4b6c-a204-3ac679d8c0d5" (UID: "8ba7753d-250d-4b6c-a204-3ac679d8c0d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:30.103763 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:30.103669    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7753d-250d-4b6c-a204-3ac679d8c0d5-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:30.808703 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:30.808664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m" event={"ID":"8ba7753d-250d-4b6c-a204-3ac679d8c0d5","Type":"ContainerDied","Data":"8533f0e5bb394b79eb03aadebde94437cf4465b0d497f55dd823c96600ecd689"}
Sep 16 18:56:30.808882 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:30.808717    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8533f0e5bb394b79eb03aadebde94437cf4465b0d497f55dd823c96600ecd689"
Sep 16 18:56:30.808882 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:30.808678    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"
Sep 16 18:56:32.345847 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:32.345815    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"]
Sep 16 18:56:32.356834 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:32.356809    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"]
Sep 16 18:56:32.368174 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:32.368142    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"]
Sep 16 18:56:41.236181 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:41.236146    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:56:47.349277 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.349237    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7"]
Sep 16 18:56:47.367047 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.366995    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b" podUID="f10b5732-a5d0-474a-8fbb-7ae3727c60bc" containerName="main" containerID="cri-o://416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a" gracePeriod=45
Sep 16 18:56:47.367726 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.367643    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-662432-29826405-pj9j7"]
Sep 16 18:56:47.381935 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.381830    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx" podUID="1737c29b-2643-4d61-8678-7222756a5a93" containerName="main" containerID="cri-o://32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191" gracePeriod=45
Sep 16 18:56:47.388780 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.388743    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q" podUID="e55c2cfb-9598-4621-adc5-a059a975a236" containerName="main" containerID="cri-o://53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc" gracePeriod=45
Sep 16 18:56:47.648000 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.647969    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:56:47.669456 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.669420    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6t95\" (UniqueName: \"kubernetes.io/projected/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-kube-api-access-k6t95\") pod \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") "
Sep 16 18:56:47.669630 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.669493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-model-cache\") pod \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") "
Sep 16 18:56:47.669630 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.669563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tls-certs\") pod \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") "
Sep 16 18:56:47.669630 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.669612    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-home\") pod \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") "
Sep 16 18:56:47.669815 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.669665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-dshm\") pod \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") "
Sep 16 18:56:47.669815 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.669780    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "f10b5732-a5d0-474a-8fbb-7ae3727c60bc" (UID: "f10b5732-a5d0-474a-8fbb-7ae3727c60bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.669815 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.669787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tmp-dir\") pod \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\" (UID: \"f10b5732-a5d0-474a-8fbb-7ae3727c60bc\") "
Sep 16 18:56:47.670050 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.670001    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f10b5732-a5d0-474a-8fbb-7ae3727c60bc" (UID: "f10b5732-a5d0-474a-8fbb-7ae3727c60bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.670188 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.670145    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-home" (OuterVolumeSpecName: "home") pod "f10b5732-a5d0-474a-8fbb-7ae3727c60bc" (UID: "f10b5732-a5d0-474a-8fbb-7ae3727c60bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.670257 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.670197    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tmp-dir\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.670257 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.670214    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-model-cache\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.671947 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.671896    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f10b5732-a5d0-474a-8fbb-7ae3727c60bc" (UID: "f10b5732-a5d0-474a-8fbb-7ae3727c60bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:47.672073 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.672054    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-kube-api-access-k6t95" (OuterVolumeSpecName: "kube-api-access-k6t95") pod "f10b5732-a5d0-474a-8fbb-7ae3727c60bc" (UID: "f10b5732-a5d0-474a-8fbb-7ae3727c60bc"). InnerVolumeSpecName "kube-api-access-k6t95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:47.672167 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.672153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-dshm" (OuterVolumeSpecName: "dshm") pod "f10b5732-a5d0-474a-8fbb-7ae3727c60bc" (UID: "f10b5732-a5d0-474a-8fbb-7ae3727c60bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.682416 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.682394    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:56:47.685636 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.685620    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:56:47.770817 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.770781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-tmp-dir\") pod \"e55c2cfb-9598-4621-adc5-a059a975a236\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") "
Sep 16 18:56:47.771022 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.770842    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-model-cache\") pod \"1737c29b-2643-4d61-8678-7222756a5a93\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") "
Sep 16 18:56:47.771022 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.770879    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-home\") pod \"1737c29b-2643-4d61-8678-7222756a5a93\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") "
Sep 16 18:56:47.771022 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.770945    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55c2cfb-9598-4621-adc5-a059a975a236-tls-certs\") pod \"e55c2cfb-9598-4621-adc5-a059a975a236\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") "
Sep 16 18:56:47.771022 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.770981    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-home\") pod \"e55c2cfb-9598-4621-adc5-a059a975a236\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") "
Sep 16 18:56:47.771272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771060    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmnrs\" (UniqueName: \"kubernetes.io/projected/e55c2cfb-9598-4621-adc5-a059a975a236-kube-api-access-gmnrs\") pod \"e55c2cfb-9598-4621-adc5-a059a975a236\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") "
Sep 16 18:56:47.771272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1737c29b-2643-4d61-8678-7222756a5a93-tls-certs\") pod \"1737c29b-2643-4d61-8678-7222756a5a93\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") "
Sep 16 18:56:47.771272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771127    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-model-cache\") pod \"e55c2cfb-9598-4621-adc5-a059a975a236\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") "
Sep 16 18:56:47.771272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771152    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-dshm\") pod \"1737c29b-2643-4d61-8678-7222756a5a93\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") "
Sep 16 18:56:47.771272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bws4n\" (UniqueName: \"kubernetes.io/projected/1737c29b-2643-4d61-8678-7222756a5a93-kube-api-access-bws4n\") pod \"1737c29b-2643-4d61-8678-7222756a5a93\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") "
Sep 16 18:56:47.771272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771216    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-tmp-dir\") pod \"1737c29b-2643-4d61-8678-7222756a5a93\" (UID: \"1737c29b-2643-4d61-8678-7222756a5a93\") "
Sep 16 18:56:47.771272 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-dshm\") pod \"e55c2cfb-9598-4621-adc5-a059a975a236\" (UID: \"e55c2cfb-9598-4621-adc5-a059a975a236\") "
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771623    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6t95\" (UniqueName: \"kubernetes.io/projected/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-kube-api-access-k6t95\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771647    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-tls-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771663    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-home\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771674    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10b5732-a5d0-474a-8fbb-7ae3727c60bc-dshm\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771118    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-model-cache" (OuterVolumeSpecName: "model-cache") pod "1737c29b-2643-4d61-8678-7222756a5a93" (UID: "1737c29b-2643-4d61-8678-7222756a5a93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771122    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e55c2cfb-9598-4621-adc5-a059a975a236" (UID: "e55c2cfb-9598-4621-adc5-a059a975a236"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771155    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-home" (OuterVolumeSpecName: "home") pod "1737c29b-2643-4d61-8678-7222756a5a93" (UID: "1737c29b-2643-4d61-8678-7222756a5a93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771417    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-home" (OuterVolumeSpecName: "home") pod "e55c2cfb-9598-4621-adc5-a059a975a236" (UID: "e55c2cfb-9598-4621-adc5-a059a975a236"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771682    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-model-cache" (OuterVolumeSpecName: "model-cache") pod "e55c2cfb-9598-4621-adc5-a059a975a236" (UID: "e55c2cfb-9598-4621-adc5-a059a975a236"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.771958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1737c29b-2643-4d61-8678-7222756a5a93" (UID: "1737c29b-2643-4d61-8678-7222756a5a93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.774356    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1737c29b-2643-4d61-8678-7222756a5a93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1737c29b-2643-4d61-8678-7222756a5a93" (UID: "1737c29b-2643-4d61-8678-7222756a5a93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.774640    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-dshm" (OuterVolumeSpecName: "dshm") pod "1737c29b-2643-4d61-8678-7222756a5a93" (UID: "1737c29b-2643-4d61-8678-7222756a5a93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.776710 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.775462    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-dshm" (OuterVolumeSpecName: "dshm") pod "e55c2cfb-9598-4621-adc5-a059a975a236" (UID: "e55c2cfb-9598-4621-adc5-a059a975a236"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.777701 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.777660    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1737c29b-2643-4d61-8678-7222756a5a93-kube-api-access-bws4n" (OuterVolumeSpecName: "kube-api-access-bws4n") pod "1737c29b-2643-4d61-8678-7222756a5a93" (UID: "1737c29b-2643-4d61-8678-7222756a5a93"). InnerVolumeSpecName "kube-api-access-bws4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:47.778190 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.778158    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55c2cfb-9598-4621-adc5-a059a975a236-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e55c2cfb-9598-4621-adc5-a059a975a236" (UID: "e55c2cfb-9598-4621-adc5-a059a975a236"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:47.780180 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.780156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55c2cfb-9598-4621-adc5-a059a975a236-kube-api-access-gmnrs" (OuterVolumeSpecName: "kube-api-access-gmnrs") pod "e55c2cfb-9598-4621-adc5-a059a975a236" (UID: "e55c2cfb-9598-4621-adc5-a059a975a236"). InnerVolumeSpecName "kube-api-access-gmnrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:47.785649 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.785622    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130a948c-0e38-4cae-bf51-c10b332bf1a5" path="/var/lib/kubelet/pods/130a948c-0e38-4cae-bf51-c10b332bf1a5/volumes"
Sep 16 18:56:47.851064 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.851023    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"]
Sep 16 18:56:47.859058 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.859021    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-662432-9g58m"]
Sep 16 18:56:47.873532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873439    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-model-cache\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.873532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873473    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-home\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.873532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873492    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55c2cfb-9598-4621-adc5-a059a975a236-tls-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.873532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873502    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-home\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.873532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873516    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmnrs\" (UniqueName: \"kubernetes.io/projected/e55c2cfb-9598-4621-adc5-a059a975a236-kube-api-access-gmnrs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.873532 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873531    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1737c29b-2643-4d61-8678-7222756a5a93-tls-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.873861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873549    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-model-cache\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.873861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873560    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-dshm\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.873861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873572    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bws4n\" (UniqueName: \"kubernetes.io/projected/1737c29b-2643-4d61-8678-7222756a5a93-kube-api-access-bws4n\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.873861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873586    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1737c29b-2643-4d61-8678-7222756a5a93-tmp-dir\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.873861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873598    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-dshm\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.873861 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.873609    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55c2cfb-9598-4621-adc5-a059a975a236-tmp-dir\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.900487 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.900445    2580 generic.go:356] "Generic (PLEG): container finished" podID="f10b5732-a5d0-474a-8fbb-7ae3727c60bc" containerID="416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a" exitCode=0
Sep 16 18:56:47.900487 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.900491    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"
Sep 16 18:56:47.900757 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.900525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b" event={"ID":"f10b5732-a5d0-474a-8fbb-7ae3727c60bc","Type":"ContainerDied","Data":"416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a"}
Sep 16 18:56:47.900757 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.900573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b" event={"ID":"f10b5732-a5d0-474a-8fbb-7ae3727c60bc","Type":"ContainerDied","Data":"612c659ad47bfa0b37e9e97c3780730db507f6d57df5bbc39dcb88b4d5c89f31"}
Sep 16 18:56:47.900757 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.900596    2580 scope.go:117] "RemoveContainer" containerID="416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a"
Sep 16 18:56:47.902037 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.902013    2580 generic.go:356] "Generic (PLEG): container finished" podID="e55c2cfb-9598-4621-adc5-a059a975a236" containerID="53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc" exitCode=0
Sep 16 18:56:47.902145 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.902060    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"
Sep 16 18:56:47.902145 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.902103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q" event={"ID":"e55c2cfb-9598-4621-adc5-a059a975a236","Type":"ContainerDied","Data":"53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc"}
Sep 16 18:56:47.902284 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.902149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q" event={"ID":"e55c2cfb-9598-4621-adc5-a059a975a236","Type":"ContainerDied","Data":"7c6ff53bdb6e9ebad03211347ac3a9a808cae38886d5e0ded4e862ac73927c4d"}
Sep 16 18:56:47.903795 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.903773    2580 generic.go:356] "Generic (PLEG): container finished" podID="1737c29b-2643-4d61-8678-7222756a5a93" containerID="32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191" exitCode=0
Sep 16 18:56:47.903899 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.903808    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"
Sep 16 18:56:47.903899 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.903806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx" event={"ID":"1737c29b-2643-4d61-8678-7222756a5a93","Type":"ContainerDied","Data":"32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191"}
Sep 16 18:56:47.904013 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.903928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx" event={"ID":"1737c29b-2643-4d61-8678-7222756a5a93","Type":"ContainerDied","Data":"4ec9225bf0f2332cea21f6f646be668d8f71e15d48cab384a59057366b006521"}
Sep 16 18:56:47.911041 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.911026    2580 scope.go:117] "RemoveContainer" containerID="416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a"
Sep 16 18:56:47.911287 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:56:47.911268    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a\": container with ID starting with 416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a not found: ID does not exist" containerID="416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a"
Sep 16 18:56:47.911379 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.911294    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a"} err="failed to get container status \"416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a\": rpc error: code = NotFound desc = could not find container \"416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a\": container with ID starting with 416905ab926ad833ca01d6e1ae22cf4a416c1f1a4922678ecdc80eb5f9c34e8a not found: ID does not exist"
Sep 16 18:56:47.911379 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.911314    2580 scope.go:117] "RemoveContainer" containerID="53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc"
Sep 16 18:56:47.918894 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.918879    2580 scope.go:117] "RemoveContainer" containerID="53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc"
Sep 16 18:56:47.919156 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:56:47.919138    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc\": container with ID starting with 53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc not found: ID does not exist" containerID="53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc"
Sep 16 18:56:47.919201 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.919160    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc"} err="failed to get container status \"53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc\": rpc error: code = NotFound desc = could not find container \"53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc\": container with ID starting with 53f71e7ecf373373f6beabcc068836c269d8951a95b8d8d6e6a0be34dbfb27cc not found: ID does not exist"
Sep 16 18:56:47.919201 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.919176    2580 scope.go:117] "RemoveContainer" containerID="32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191"
Sep 16 18:56:47.926722 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.926699    2580 scope.go:117] "RemoveContainer" containerID="32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191"
Sep 16 18:56:47.926978 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:56:47.926955    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191\": container with ID starting with 32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191 not found: ID does not exist" containerID="32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191"
Sep 16 18:56:47.927034 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.926981    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191"} err="failed to get container status \"32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191\": rpc error: code = NotFound desc = could not find container \"32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191\": container with ID starting with 32c060cc76e0224d033c4c5d8ee80cb8f5d5d84d3d3011ea61b4efa018c95191 not found: ID does not exist"
Sep 16 18:56:47.931537 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.931520    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"]
Sep 16 18:56:47.934059 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.934040    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-662432/distinct-2-w4-662432-kserve-7789d6dd47-s6q9b"]
Sep 16 18:56:47.959420 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.959387    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"]
Sep 16 18:56:47.962633 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.962608    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-662432/simulator-w4-662432-kserve-c7b445cb8-7zw4q"]
Sep 16 18:56:47.977236 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.977211    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"]
Sep 16 18:56:47.979232 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:47.979210    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-662432/premium-w4-662432-kserve-5b5bffbfc-gkfdx"]
Sep 16 18:56:49.783870 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:49.783837    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1737c29b-2643-4d61-8678-7222756a5a93" path="/var/lib/kubelet/pods/1737c29b-2643-4d61-8678-7222756a5a93/volumes"
Sep 16 18:56:49.784255 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:49.784174    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba7753d-250d-4b6c-a204-3ac679d8c0d5" path="/var/lib/kubelet/pods/8ba7753d-250d-4b6c-a204-3ac679d8c0d5/volumes"
Sep 16 18:56:49.784537 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:49.784524    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e55c2cfb-9598-4621-adc5-a059a975a236" path="/var/lib/kubelet/pods/e55c2cfb-9598-4621-adc5-a059a975a236/volumes"
Sep 16 18:56:49.785138 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:56:49.785124    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f10b5732-a5d0-474a-8fbb-7ae3727c60bc" path="/var/lib/kubelet/pods/f10b5732-a5d0-474a-8fbb-7ae3727c60bc/volumes"
Sep 16 18:57:03.067484 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:03.067447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:57:13.439680 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:13.439633    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:57:23.003807 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:23.003773    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:57:31.974080 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.973998    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"]
Sep 16 18:57:31.974442 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974373    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1737c29b-2643-4d61-8678-7222756a5a93" containerName="main"
Sep 16 18:57:31.974442 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974386    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1737c29b-2643-4d61-8678-7222756a5a93" containerName="main"
Sep 16 18:57:31.974442 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974395    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba7753d-250d-4b6c-a204-3ac679d8c0d5" containerName="revoke-keys"
Sep 16 18:57:31.974442 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974401    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba7753d-250d-4b6c-a204-3ac679d8c0d5" containerName="revoke-keys"
Sep 16 18:57:31.974442 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974420    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f10b5732-a5d0-474a-8fbb-7ae3727c60bc" containerName="main"
Sep 16 18:57:31.974442 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974427    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f10b5732-a5d0-474a-8fbb-7ae3727c60bc" containerName="main"
Sep 16 18:57:31.974442 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974437    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55c2cfb-9598-4621-adc5-a059a975a236" containerName="main"
Sep 16 18:57:31.974442 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974443    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55c2cfb-9598-4621-adc5-a059a975a236" containerName="main"
Sep 16 18:57:31.974697 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974513    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb167c06-6fa4-4437-9994-709b682d6634" containerName="istio-proxy"
Sep 16 18:57:31.974697 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974521    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1737c29b-2643-4d61-8678-7222756a5a93" containerName="main"
Sep 16 18:57:31.974697 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974527    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba7753d-250d-4b6c-a204-3ac679d8c0d5" containerName="revoke-keys"
Sep 16 18:57:31.974697 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974534    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e55c2cfb-9598-4621-adc5-a059a975a236" containerName="main"
Sep 16 18:57:31.974697 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.974539    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f10b5732-a5d0-474a-8fbb-7ae3727c60bc" containerName="main"
Sep 16 18:57:31.977714 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.977699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:31.982475 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.982447    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f9c9b8\"/\"kube-root-ca.crt\""
Sep 16 18:57:31.982611 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.982511    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f9c9b8\"/\"default-dockercfg-pr5sv\""
Sep 16 18:57:31.982611 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.982454    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f9c9b8\"/\"openshift-service-ca.crt\""
Sep 16 18:57:31.982611 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.982542    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f9c9b8\"/\"sub-test-model-f9c9b8-kserve-self-signed-certs\""
Sep 16 18:57:31.997435 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:31.997415    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"]
Sep 16 18:57:32.072257 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.072229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tmp-dir\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.072438 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.072269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tls-certs\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.072438 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.072358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-home\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.072438 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.072383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-dshm\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.072438 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.072409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-model-cache\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.072583 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.072450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv2kx\" (UniqueName: \"kubernetes.io/projected/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-kube-api-access-bv2kx\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.173261 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.173230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2kx\" (UniqueName: \"kubernetes.io/projected/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-kube-api-access-bv2kx\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.173455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.173279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tmp-dir\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.173455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.173301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tls-certs\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.173455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.173362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-home\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.173455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.173380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-dshm\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.173455 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.173403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-model-cache\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.173722 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.173685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tmp-dir\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.173778 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.173722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-model-cache\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.173826 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.173772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-home\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.175690 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.175666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-dshm\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.175901 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.175881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tls-certs\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.185691 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.185669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2kx\" (UniqueName: \"kubernetes.io/projected/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-kube-api-access-bv2kx\") pod \"sub-test-model-f9c9b8-kserve-7f9d456597-skmdg\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") " pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.286640 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.286608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:32.450869 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.450833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:57:32.564388 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:32.564361    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"]
Sep 16 18:57:32.566457 ip-10-0-141-16 kubenswrapper[2580]: W0916 18:57:32.566431    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4c15ee7_50ee_4d9d_aa2a_dd02dcfb7b40.slice/crio-dfaa8642e4cf0b70ff15b143f573a1ab6940c4ecf07cb4301812c720a4bb8893 WatchSource:0}: Error finding container dfaa8642e4cf0b70ff15b143f573a1ab6940c4ecf07cb4301812c720a4bb8893: Status 404 returned error can't find the container with id dfaa8642e4cf0b70ff15b143f573a1ab6940c4ecf07cb4301812c720a4bb8893
Sep 16 18:57:33.143127 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:33.143092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg" event={"ID":"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40","Type":"ContainerStarted","Data":"8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74"}
Sep 16 18:57:33.143127 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:33.143134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg" event={"ID":"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40","Type":"ContainerStarted","Data":"dfaa8642e4cf0b70ff15b143f573a1ab6940c4ecf07cb4301812c720a4bb8893"}
Sep 16 18:57:33.179166 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:33.179100    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg" podStartSLOduration=2.179081815 podStartE2EDuration="2.179081815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:33.175270623 +0000 UTC m=+1795.884510016" watchObservedRunningTime="2026-09-16 18:57:33.179081815 +0000 UTC m=+1795.888321208"
Sep 16 18:57:37.884641 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:37.884608    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:57:37.889069 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:37.889032    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 18:57:38.204500 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:38.204422    2580 scope.go:117] "RemoveContainer" containerID="d6b756b9721227f434b9693e77994cd773aa29373db77bf873983a86ad8f658b"
Sep 16 18:57:38.212534 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:38.212511    2580 scope.go:117] "RemoveContainer" containerID="bf13246317654f6ae9a7981a22f2b0df086eb92b1a8936598c04510fc1730d60"
Sep 16 18:57:41.532433 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:41.532402    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:57:42.287205 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:42.287172    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:42.287441 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:42.287217    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:42.299878 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:42.299856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:43.213925 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:43.213895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:57:59.947663 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:57:59.947628    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:58:23.450564 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:23.450528    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:58:38.839983 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:38.839944    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:58:40.662271 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:40.662238    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"]
Sep 16 18:58:50.283276 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:50.283240    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 18:58:55.680817 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:55.680768    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg" podUID="f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" containerName="main" containerID="cri-o://8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74" gracePeriod=45
Sep 16 18:58:55.931323 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:55.931271    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:58:56.116022 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.115981    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-model-cache\") pod \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") "
Sep 16 18:58:56.116228 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.116069    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-dshm\") pod \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") "
Sep 16 18:58:56.116228 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.116121    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tls-certs\") pod \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") "
Sep 16 18:58:56.116228 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.116180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tmp-dir\") pod \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") "
Sep 16 18:58:56.116228 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.116214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-home\") pod \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") "
Sep 16 18:58:56.116487 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.116249    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv2kx\" (UniqueName: \"kubernetes.io/projected/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-kube-api-access-bv2kx\") pod \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\" (UID: \"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40\") "
Sep 16 18:58:56.116487 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.116295    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" (UID: "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:56.116598 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.116553    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" (UID: "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:56.116598 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.116566    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-home" (OuterVolumeSpecName: "home") pod "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" (UID: "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:56.116598 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.116587    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-model-cache\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:58:56.118365 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.118344    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" (UID: "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:56.118453 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.118408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-kube-api-access-bv2kx" (OuterVolumeSpecName: "kube-api-access-bv2kx") pod "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" (UID: "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40"). InnerVolumeSpecName "kube-api-access-bv2kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:56.118501 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.118483    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-dshm" (OuterVolumeSpecName: "dshm") pod "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" (UID: "f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:56.217961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.217869    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tmp-dir\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:58:56.217961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.217902    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-home\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:58:56.217961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.217911    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv2kx\" (UniqueName: \"kubernetes.io/projected/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-kube-api-access-bv2kx\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:58:56.217961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.217920    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-dshm\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:58:56.217961 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.217929    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40-tls-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 18:58:56.578270 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.578229    2580 generic.go:356] "Generic (PLEG): container finished" podID="f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" containerID="8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74" exitCode=0
Sep 16 18:58:56.578270 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.578275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg" event={"ID":"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40","Type":"ContainerDied","Data":"8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74"}
Sep 16 18:58:56.578555 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.578278    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"
Sep 16 18:58:56.578555 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.578308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg" event={"ID":"f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40","Type":"ContainerDied","Data":"dfaa8642e4cf0b70ff15b143f573a1ab6940c4ecf07cb4301812c720a4bb8893"}
Sep 16 18:58:56.578555 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.578324    2580 scope.go:117] "RemoveContainer" containerID="8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74"
Sep 16 18:58:56.587419 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.587399    2580 scope.go:117] "RemoveContainer" containerID="8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74"
Sep 16 18:58:56.587703 ip-10-0-141-16 kubenswrapper[2580]: E0916 18:58:56.587684    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74\": container with ID starting with 8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74 not found: ID does not exist" containerID="8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74"
Sep 16 18:58:56.587772 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.587719    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74"} err="failed to get container status \"8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74\": rpc error: code = NotFound desc = could not find container \"8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74\": container with ID starting with 8a121ba7deb69a9ee2c1cef4c8f3a2b89e5b767a06a6ecca9e97f46a42c39b74 not found: ID does not exist"
Sep 16 18:58:56.606996 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.606964    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"]
Sep 16 18:58:56.615357 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:56.615320    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f9c9b8/sub-test-model-f9c9b8-kserve-7f9d456597-skmdg"]
Sep 16 18:58:57.788165 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:58:57.788128    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" path="/var/lib/kubelet/pods/f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40/volumes"
Sep 16 18:59:03.970719 ip-10-0-141-16 kubenswrapper[2580]: I0916 18:59:03.970634    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:00:00.170005 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.169973    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6"]
Sep 16 19:00:00.170488 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.170312    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" containerName="main"
Sep 16 19:00:00.170488 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.170324    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" containerName="main"
Sep 16 19:00:00.170488 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.170427    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4c15ee7-50ee-4d9d-aa2a-dd02dcfb7b40" containerName="main"
Sep 16 19:00:00.173346 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.173314    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm"]
Sep 16 19:00:00.173468 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.173446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6"
Sep 16 19:00:00.177009 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.176992    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm"
Sep 16 19:00:00.177383 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.177365    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 19:00:00.184716 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.184696    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6"
Sep 16 19:00:00.190182 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.190166    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm"
Sep 16 19:00:00.197181 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.197157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6"]
Sep 16 19:00:00.199727 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.199707    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm"]
Sep 16 19:00:00.473915 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.473876    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6"]
Sep 16 19:00:00.476159 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:00:00.476130    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edb2656_2aa2_4b54_8191_6c9849cecc0d.slice/crio-2fc738c3c8a3483b659b7a8b3dc9f6158c029b632db0f6d6634b070638d8b546 WatchSource:0}: Error finding container 2fc738c3c8a3483b659b7a8b3dc9f6158c029b632db0f6d6634b070638d8b546: Status 404 returned error can't find the container with id 2fc738c3c8a3483b659b7a8b3dc9f6158c029b632db0f6d6634b070638d8b546
Sep 16 19:00:00.493433 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.493392    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm"]
Sep 16 19:00:00.495742 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:00:00.495715    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54b1bc4_2efb_40c3_bbb9_6e8f4af3da0d.slice/crio-d0ddc7c42df38c38371c1e812cf460e3b821bd2cf2b6517317388977457ba4df WatchSource:0}: Error finding container d0ddc7c42df38c38371c1e812cf460e3b821bd2cf2b6517317388977457ba4df: Status 404 returned error can't find the container with id d0ddc7c42df38c38371c1e812cf460e3b821bd2cf2b6517317388977457ba4df
Sep 16 19:00:00.905640 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.905604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm" event={"ID":"c54b1bc4-2efb-40c3-bbb9-6e8f4af3da0d","Type":"ContainerStarted","Data":"33ada65f72d691c947ab10e91f52ac89ac8b0b05c2848c98b58be4726fe95438"}
Sep 16 19:00:00.905829 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.905657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm" event={"ID":"c54b1bc4-2efb-40c3-bbb9-6e8f4af3da0d","Type":"ContainerStarted","Data":"d0ddc7c42df38c38371c1e812cf460e3b821bd2cf2b6517317388977457ba4df"}
Sep 16 19:00:00.907180 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.907155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6" event={"ID":"7edb2656-2aa2-4b54-8191-6c9849cecc0d","Type":"ContainerStarted","Data":"9d0039c76670907f75421507c4808391be13945bc93248fc1dabea94e09669eb"}
Sep 16 19:00:00.907293 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.907199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6" event={"ID":"7edb2656-2aa2-4b54-8191-6c9849cecc0d","Type":"ContainerStarted","Data":"2fc738c3c8a3483b659b7a8b3dc9f6158c029b632db0f6d6634b070638d8b546"}
Sep 16 19:00:00.937116 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.937077    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm" podStartSLOduration=0.937061989 podStartE2EDuration="937.061989ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:00.935469339 +0000 UTC m=+1943.644708732" watchObservedRunningTime="2026-09-16 19:00:00.937061989 +0000 UTC m=+1943.646301381"
Sep 16 19:00:00.963413 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:00.963375    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6" podStartSLOduration=0.963361666 podStartE2EDuration="963.361666ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:00.962434749 +0000 UTC m=+1943.671674140" watchObservedRunningTime="2026-09-16 19:00:00.963361666 +0000 UTC m=+1943.672601073"
Sep 16 19:00:01.913867 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:01.913826    2580 generic.go:356] "Generic (PLEG): container finished" podID="7edb2656-2aa2-4b54-8191-6c9849cecc0d" containerID="9d0039c76670907f75421507c4808391be13945bc93248fc1dabea94e09669eb" exitCode=0
Sep 16 19:00:01.913867 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:01.913856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6" event={"ID":"7edb2656-2aa2-4b54-8191-6c9849cecc0d","Type":"ContainerDied","Data":"9d0039c76670907f75421507c4808391be13945bc93248fc1dabea94e09669eb"}
Sep 16 19:00:01.915750 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:01.915721    2580 generic.go:356] "Generic (PLEG): container finished" podID="c54b1bc4-2efb-40c3-bbb9-6e8f4af3da0d" containerID="33ada65f72d691c947ab10e91f52ac89ac8b0b05c2848c98b58be4726fe95438" exitCode=0
Sep 16 19:00:01.915882 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:01.915766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm" event={"ID":"c54b1bc4-2efb-40c3-bbb9-6e8f4af3da0d","Type":"ContainerDied","Data":"33ada65f72d691c947ab10e91f52ac89ac8b0b05c2848c98b58be4726fe95438"}
Sep 16 19:00:03.068867 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:03.068843    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm"
Sep 16 19:00:03.071933 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:03.071910    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6"
Sep 16 19:00:03.926279 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:03.926253    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm"
Sep 16 19:00:03.926279 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:03.926262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm" event={"ID":"c54b1bc4-2efb-40c3-bbb9-6e8f4af3da0d","Type":"ContainerDied","Data":"d0ddc7c42df38c38371c1e812cf460e3b821bd2cf2b6517317388977457ba4df"}
Sep 16 19:00:03.926536 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:03.926310    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0ddc7c42df38c38371c1e812cf460e3b821bd2cf2b6517317388977457ba4df"
Sep 16 19:00:03.927869 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:03.927836    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6"
Sep 16 19:00:03.927997 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:03.927887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826420-67hp6" event={"ID":"7edb2656-2aa2-4b54-8191-6c9849cecc0d","Type":"ContainerDied","Data":"2fc738c3c8a3483b659b7a8b3dc9f6158c029b632db0f6d6634b070638d8b546"}
Sep 16 19:00:03.927997 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:03.927931    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc738c3c8a3483b659b7a8b3dc9f6158c029b632db0f6d6634b070638d8b546"
Sep 16 19:00:10.579013 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:10.578979    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:00:37.136025 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:37.135987    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:00:38.265273 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:38.265241    2580 scope.go:117] "RemoveContainer" containerID="fcb92e19e7aa205167df0e267ccb7b4aa4b3398cb96bf534a32ac61c0df02464"
Sep 16 19:00:51.958477 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:00:51.958438    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:01:02.690896 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.690854    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"]
Sep 16 19:01:02.691404 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.691385    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7edb2656-2aa2-4b54-8191-6c9849cecc0d" containerName="cleanup"
Sep 16 19:01:02.691495 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.691407    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7edb2656-2aa2-4b54-8191-6c9849cecc0d" containerName="cleanup"
Sep 16 19:01:02.691495 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.691434    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c54b1bc4-2efb-40c3-bbb9-6e8f4af3da0d" containerName="cleanup"
Sep 16 19:01:02.691495 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.691444    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c54b1bc4-2efb-40c3-bbb9-6e8f4af3da0d" containerName="cleanup"
Sep 16 19:01:02.691652 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.691542    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c54b1bc4-2efb-40c3-bbb9-6e8f4af3da0d" containerName="cleanup"
Sep 16 19:01:02.691652 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.691559    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7edb2656-2aa2-4b54-8191-6c9849cecc0d" containerName="cleanup"
Sep 16 19:01:02.694788 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.694767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.714307 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.714273    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:01:02.714446 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.714427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6a532e-openshift-default-dockercfg-ht7hb\""
Sep 16 19:01:02.758959 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.758932    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"]
Sep 16 19:01:02.813786 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.813749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-envoy\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.813946 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.813796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-data\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.813946 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.813853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee750069-54c9-439b-84ee-0c96942cd03e-istio-podinfo\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.813946 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.813869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-istio-token\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.813946 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.813904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb8h4\" (UniqueName: \"kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-kube-api-access-qb8h4\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.813946 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.813936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-credential-socket\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.814183 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.813955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-socket\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.814183 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.814058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee750069-54c9-439b-84ee-0c96942cd03e-istiod-ca-cert\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.814183 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.814083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-certs\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915116 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee750069-54c9-439b-84ee-0c96942cd03e-istiod-ca-cert\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915116 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-certs\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915315 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-envoy\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915315 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-data\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915315 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee750069-54c9-439b-84ee-0c96942cd03e-istio-podinfo\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915315 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-istio-token\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915315 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8h4\" (UniqueName: \"kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-kube-api-access-qb8h4\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915564 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-credential-socket\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915564 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-socket\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915963 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-credential-socket\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915963 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-socket\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.915963 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-data\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.916158 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915978    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-certs\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.916158 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.915961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee750069-54c9-439b-84ee-0c96942cd03e-istiod-ca-cert\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.917577 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.917555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-envoy\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.917823 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.917805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee750069-54c9-439b-84ee-0c96942cd03e-istio-podinfo\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.936053 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.936033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8h4\" (UniqueName: \"kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-kube-api-access-qb8h4\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:02.937440 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:02.937420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-istio-token\") pod \"e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") " pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:03.005235 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:03.005160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:03.319773 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:03.319743    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"]
Sep 16 19:01:03.323359 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:01:03.323309    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee750069_54c9_439b_84ee_0c96942cd03e.slice/crio-d887a0219f889b431866bd0fafc36bf337975f7298a20a38c7e9e2e25bbf943d WatchSource:0}: Error finding container d887a0219f889b431866bd0fafc36bf337975f7298a20a38c7e9e2e25bbf943d: Status 404 returned error can't find the container with id d887a0219f889b431866bd0fafc36bf337975f7298a20a38c7e9e2e25bbf943d
Sep 16 19:01:03.325382 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:03.325353    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 19:01:03.325445 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:03.325415    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 19:01:03.325483 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:03.325444    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 19:01:04.225725 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:04.225687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" event={"ID":"ee750069-54c9-439b-84ee-0c96942cd03e","Type":"ContainerStarted","Data":"f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e"}
Sep 16 19:01:04.226197 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:04.225784    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" event={"ID":"ee750069-54c9-439b-84ee-0c96942cd03e","Type":"ContainerStarted","Data":"d887a0219f889b431866bd0fafc36bf337975f7298a20a38c7e9e2e25bbf943d"}
Sep 16 19:01:04.232401 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:04.232376    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:01:04.284111 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:04.284058    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" podStartSLOduration=2.284044156 podStartE2EDuration="2.284044156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:04.273419963 +0000 UTC m=+2006.982659356" watchObservedRunningTime="2026-09-16 19:01:04.284044156 +0000 UTC m=+2006.993283548"
Sep 16 19:01:05.006191 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:05.006156    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:05.010972 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:05.010948    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:05.231062 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:05.231029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:05.231851 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:05.231835    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:14.272197 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:14.272160    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:01:25.341519 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:25.341431    2580 generic.go:356] "Generic (PLEG): container finished" podID="ee750069-54c9-439b-84ee-0c96942cd03e" containerID="f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e" exitCode=0
Sep 16 19:01:25.341519 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:25.341501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" event={"ID":"ee750069-54c9-439b-84ee-0c96942cd03e","Type":"ContainerDied","Data":"f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e"}
Sep 16 19:01:25.342056 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:25.341905    2580 scope.go:117] "RemoveContainer" containerID="f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e"
Sep 16 19:01:25.342961 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:25.342934    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 19:01:25.343020 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:25.343002    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 19:01:25.343081 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:25.343033    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 19:01:26.349400 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:26.349360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" event={"ID":"ee750069-54c9-439b-84ee-0c96942cd03e","Type":"ContainerStarted","Data":"33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4"}
Sep 16 19:01:27.005999 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:27.005965    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:27.006222 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:27.006199    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 16 19:01:27.006279 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:27.006247    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 16 19:01:28.006391 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:28.006355    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 16 19:01:28.006790 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:28.006413    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 16 19:01:29.006362 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:29.006311    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 16 19:01:29.006534 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:29.006396    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 16 19:01:30.006229 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:30.006193    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 16 19:01:30.006451 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:30.006257    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 16 19:01:31.006560 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:31.006512    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 16 19:01:31.007045 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:31.006577    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 16 19:01:32.010463 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:32.010421    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:32.010851 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:32.010690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:32.011444 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:32.011424    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:01:35.220543 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.220511    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"]
Sep 16 19:01:35.222842 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.222823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:01:35.233371 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.233349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a0f152-dockercfg-mjwjj\""
Sep 16 19:01:35.233476 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.233380    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a0f152\""
Sep 16 19:01:35.247248 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.247225    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"]
Sep 16 19:01:35.299883 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.299857    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"]
Sep 16 19:01:35.302104 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.302089    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:01:35.334867 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.334836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed4bd4e-5a32-4652-a0a3-941f7824239f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9\" (UID: \"bed4bd4e-5a32-4652-a0a3-941f7824239f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:01:35.335031 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.334908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12a3805-161e-4080-a16d-17913321de50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp\" (UID: \"e12a3805-161e-4080-a16d-17913321de50\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:01:35.335031 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.334982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcckb\" (UniqueName: \"kubernetes.io/projected/bed4bd4e-5a32-4652-a0a3-941f7824239f-kube-api-access-fcckb\") pod \"payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9\" (UID: \"bed4bd4e-5a32-4652-a0a3-941f7824239f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:01:35.335156 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.335121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpmfs\" (UniqueName: \"kubernetes.io/projected/e12a3805-161e-4080-a16d-17913321de50-kube-api-access-tpmfs\") pod \"payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp\" (UID: \"e12a3805-161e-4080-a16d-17913321de50\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:01:35.335241 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.335211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"]
Sep 16 19:01:35.435758 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.435724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpmfs\" (UniqueName: \"kubernetes.io/projected/e12a3805-161e-4080-a16d-17913321de50-kube-api-access-tpmfs\") pod \"payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp\" (UID: \"e12a3805-161e-4080-a16d-17913321de50\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:01:35.435931 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.435769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed4bd4e-5a32-4652-a0a3-941f7824239f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9\" (UID: \"bed4bd4e-5a32-4652-a0a3-941f7824239f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:01:35.435931 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.435806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12a3805-161e-4080-a16d-17913321de50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp\" (UID: \"e12a3805-161e-4080-a16d-17913321de50\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:01:35.435931 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.435843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcckb\" (UniqueName: \"kubernetes.io/projected/bed4bd4e-5a32-4652-a0a3-941f7824239f-kube-api-access-fcckb\") pod \"payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9\" (UID: \"bed4bd4e-5a32-4652-a0a3-941f7824239f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:01:35.436571 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.436549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12a3805-161e-4080-a16d-17913321de50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp\" (UID: \"e12a3805-161e-4080-a16d-17913321de50\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:01:35.436628 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.436584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed4bd4e-5a32-4652-a0a3-941f7824239f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9\" (UID: \"bed4bd4e-5a32-4652-a0a3-941f7824239f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:01:35.451166 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.451138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcckb\" (UniqueName: \"kubernetes.io/projected/bed4bd4e-5a32-4652-a0a3-941f7824239f-kube-api-access-fcckb\") pod \"payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9\" (UID: \"bed4bd4e-5a32-4652-a0a3-941f7824239f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:01:35.451525 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.451500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpmfs\" (UniqueName: \"kubernetes.io/projected/e12a3805-161e-4080-a16d-17913321de50-kube-api-access-tpmfs\") pod \"payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp\" (UID: \"e12a3805-161e-4080-a16d-17913321de50\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:01:35.533203 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.533164    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:01:35.612958 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.612921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:01:35.818762 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.818737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"]
Sep 16 19:01:35.821053 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:01:35.821026    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed4bd4e_5a32_4652_a0a3_941f7824239f.slice/crio-0d18b9c455314a7418b3acb5a256979672c64606623561f389320841619f3514 WatchSource:0}: Error finding container 0d18b9c455314a7418b3acb5a256979672c64606623561f389320841619f3514: Status 404 returned error can't find the container with id 0d18b9c455314a7418b3acb5a256979672c64606623561f389320841619f3514
Sep 16 19:01:35.910801 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:35.910774    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"]
Sep 16 19:01:35.913545 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:01:35.913520    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12a3805_161e_4080_a16d_17913321de50.slice/crio-9c8b9e956e769b25ab5e79257b07f22bfd2b9bc7bea16272f16a077e82e4ebb6 WatchSource:0}: Error finding container 9c8b9e956e769b25ab5e79257b07f22bfd2b9bc7bea16272f16a077e82e4ebb6: Status 404 returned error can't find the container with id 9c8b9e956e769b25ab5e79257b07f22bfd2b9bc7bea16272f16a077e82e4ebb6
Sep 16 19:01:36.410350 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:36.410305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9" event={"ID":"bed4bd4e-5a32-4652-a0a3-941f7824239f","Type":"ContainerStarted","Data":"7352dfefc650292c31ddd318b5d57dc8c7c0a55082d1a22d40286c40452a91b3"}
Sep 16 19:01:36.410834 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:36.410425    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:01:36.410834 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:36.410454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9" event={"ID":"bed4bd4e-5a32-4652-a0a3-941f7824239f","Type":"ContainerStarted","Data":"0d18b9c455314a7418b3acb5a256979672c64606623561f389320841619f3514"}
Sep 16 19:01:36.411929 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:36.411901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp" event={"ID":"e12a3805-161e-4080-a16d-17913321de50","Type":"ContainerStarted","Data":"234a7a35466fcc7eddd39c4dcac1373b5ad2fdd244ae7f1b8eef22f3546b58d0"}
Sep 16 19:01:36.412067 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:36.411947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp" event={"ID":"e12a3805-161e-4080-a16d-17913321de50","Type":"ContainerStarted","Data":"9c8b9e956e769b25ab5e79257b07f22bfd2b9bc7bea16272f16a077e82e4ebb6"}
Sep 16 19:01:36.412067 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:36.412025    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:01:36.450268 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:36.450222    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9" podStartSLOduration=1.4502078520000001 podStartE2EDuration="1.450207852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:36.44850921 +0000 UTC m=+2039.157748603" watchObservedRunningTime="2026-09-16 19:01:36.450207852 +0000 UTC m=+2039.159447287"
Sep 16 19:01:36.494432 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:36.494389    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp" podStartSLOduration=1.49437453 podStartE2EDuration="1.49437453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:36.493026208 +0000 UTC m=+2039.202265600" watchObservedRunningTime="2026-09-16 19:01:36.49437453 +0000 UTC m=+2039.203613934"
Sep 16 19:01:38.285513 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:38.285482    2580 scope.go:117] "RemoveContainer" containerID="9f2910b71f379b7c317b35c90d1b6fbd2a002ea08fdf87684a146ab882d80446"
Sep 16 19:01:38.295563 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:38.295460    2580 scope.go:117] "RemoveContainer" containerID="9c6b7cba5abd1049077c15ea11b0354695fec618fe47b9ee0167490eca05ec70"
Sep 16 19:01:38.304057 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:38.304036    2580 scope.go:117] "RemoveContainer" containerID="4e0d3f59684c2d6bcee1e5c36660cff29c43aab496cf222f85269ae9a9b00a9a"
Sep 16 19:01:38.360912 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:38.360871    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:01:47.420185 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:47.420149    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:01:47.420646 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:47.420352    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:01:55.656061 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:01:55.656030    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp_e12a3805-161e-4080-a16d-17913321de50/payload-processing/0.log"
Sep 16 19:02:37.921959 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:02:37.921933    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 19:02:37.923837 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:02:37.923817    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 19:02:38.229959 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:02:38.229877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:02:38.332376 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:02:38.332325    2580 scope.go:117] "RemoveContainer" containerID="1d482e5f0ef619a2f46b27f57b9453295ab72545b7ad47fae4859f7809933a25"
Sep 16 19:02:38.340995 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:02:38.340975    2580 scope.go:117] "RemoveContainer" containerID="7199a3b562281c1b39d1cc5c8416b083c08c39a3ea6dd923bce559c8255f759e"
Sep 16 19:02:51.236799 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:02:51.236761    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:02:57.495726 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:02:57.495698    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp_e12a3805-161e-4080-a16d-17913321de50/payload-processing/0.log"
Sep 16 19:03:06.143768 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.143734    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"]
Sep 16 19:03:06.147317 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.147300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:06.150391 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.150371    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-0e536c-dockercfg-z9rwk\""
Sep 16 19:03:06.150516 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.150441    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-0e536c\""
Sep 16 19:03:06.159222 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.159195    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"]
Sep 16 19:03:06.205374 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.205342    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"]
Sep 16 19:03:06.208806 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.208789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:06.220550 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.220527    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"]
Sep 16 19:03:06.236850 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.236824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5kn\" (UniqueName: \"kubernetes.io/projected/f3ee7355-9180-42af-acd6-92a8c524dd5c-kube-api-access-ww5kn\") pod \"payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf\" (UID: \"f3ee7355-9180-42af-acd6-92a8c524dd5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:06.236946 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.236870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ee7355-9180-42af-acd6-92a8c524dd5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf\" (UID: \"f3ee7355-9180-42af-acd6-92a8c524dd5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:06.236946 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.236893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0598979-4760-4894-8551-77deae04f845-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg\" (UID: \"f0598979-4760-4894-8551-77deae04f845\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:06.236946 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.236924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78hwt\" (UniqueName: \"kubernetes.io/projected/f0598979-4760-4894-8551-77deae04f845-kube-api-access-78hwt\") pod \"payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg\" (UID: \"f0598979-4760-4894-8551-77deae04f845\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:06.338154 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.338116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5kn\" (UniqueName: \"kubernetes.io/projected/f3ee7355-9180-42af-acd6-92a8c524dd5c-kube-api-access-ww5kn\") pod \"payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf\" (UID: \"f3ee7355-9180-42af-acd6-92a8c524dd5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:06.338396 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.338174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ee7355-9180-42af-acd6-92a8c524dd5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf\" (UID: \"f3ee7355-9180-42af-acd6-92a8c524dd5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:06.338396 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.338197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0598979-4760-4894-8551-77deae04f845-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg\" (UID: \"f0598979-4760-4894-8551-77deae04f845\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:06.338396 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.338217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hwt\" (UniqueName: \"kubernetes.io/projected/f0598979-4760-4894-8551-77deae04f845-kube-api-access-78hwt\") pod \"payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg\" (UID: \"f0598979-4760-4894-8551-77deae04f845\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:06.339026 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.339000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0598979-4760-4894-8551-77deae04f845-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg\" (UID: \"f0598979-4760-4894-8551-77deae04f845\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:06.339125 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.339001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ee7355-9180-42af-acd6-92a8c524dd5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf\" (UID: \"f3ee7355-9180-42af-acd6-92a8c524dd5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:06.348963 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.348935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5kn\" (UniqueName: \"kubernetes.io/projected/f3ee7355-9180-42af-acd6-92a8c524dd5c-kube-api-access-ww5kn\") pod \"payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf\" (UID: \"f3ee7355-9180-42af-acd6-92a8c524dd5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:06.349050 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.348968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78hwt\" (UniqueName: \"kubernetes.io/projected/f0598979-4760-4894-8551-77deae04f845-kube-api-access-78hwt\") pod \"payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg\" (UID: \"f0598979-4760-4894-8551-77deae04f845\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:06.458048 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.457960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:06.519294 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.519104    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:06.736829 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.736799    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"]
Sep 16 19:03:06.738919 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:03:06.738896    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ee7355_9180_42af_acd6_92a8c524dd5c.slice/crio-608571a7e27894723ed4e6b61176991b948d8972060526be715013a41b1020cd WatchSource:0}: Error finding container 608571a7e27894723ed4e6b61176991b948d8972060526be715013a41b1020cd: Status 404 returned error can't find the container with id 608571a7e27894723ed4e6b61176991b948d8972060526be715013a41b1020cd
Sep 16 19:03:06.799802 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.799769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"]
Sep 16 19:03:06.801809 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:03:06.801782    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0598979_4760_4894_8551_77deae04f845.slice/crio-b4559591fd356ef0792ca15e3703f6ca4352a3019e5e03a3c1914eb8a1861c1a WatchSource:0}: Error finding container b4559591fd356ef0792ca15e3703f6ca4352a3019e5e03a3c1914eb8a1861c1a: Status 404 returned error can't find the container with id b4559591fd356ef0792ca15e3703f6ca4352a3019e5e03a3c1914eb8a1861c1a
Sep 16 19:03:06.868774 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.868747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf" event={"ID":"f3ee7355-9180-42af-acd6-92a8c524dd5c","Type":"ContainerStarted","Data":"1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012"}
Sep 16 19:03:06.868928 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.868817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf" event={"ID":"f3ee7355-9180-42af-acd6-92a8c524dd5c","Type":"ContainerStarted","Data":"608571a7e27894723ed4e6b61176991b948d8972060526be715013a41b1020cd"}
Sep 16 19:03:06.868928 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.868848    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:06.870676 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.870647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg" event={"ID":"f0598979-4760-4894-8551-77deae04f845","Type":"ContainerStarted","Data":"b4559591fd356ef0792ca15e3703f6ca4352a3019e5e03a3c1914eb8a1861c1a"}
Sep 16 19:03:06.895351 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:06.895271    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf" podStartSLOduration=0.895250368 podStartE2EDuration="895.250368ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:06.892768935 +0000 UTC m=+2129.602008328" watchObservedRunningTime="2026-09-16 19:03:06.895250368 +0000 UTC m=+2129.604489761"
Sep 16 19:03:07.878707 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:07.878657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg" event={"ID":"f0598979-4760-4894-8551-77deae04f845","Type":"ContainerStarted","Data":"590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f"}
Sep 16 19:03:07.899823 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:07.899765    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg" podStartSLOduration=1.899745604 podStartE2EDuration="1.899745604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:07.89728496 +0000 UTC m=+2130.606524350" watchObservedRunningTime="2026-09-16 19:03:07.899745604 +0000 UTC m=+2130.608984998"
Sep 16 19:03:08.883559 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:08.883527    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:11.432797 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:11.432758    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:03:17.880902 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:17.880872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:19.890751 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:19.890720    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:21.879737 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:21.879704    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"]
Sep 16 19:03:21.883219 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:21.883200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"
Sep 16 19:03:21.886016 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:21.885991    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:03:21.886133 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:21.886053    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 19:03:21.903403 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:21.903359    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"]
Sep 16 19:03:21.985604 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:21.985564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209e47ed-5313-484a-a31c-70c35dab9738-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44\" (UID: \"209e47ed-5313-484a-a31c-70c35dab9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"
Sep 16 19:03:22.086648 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:22.086610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209e47ed-5313-484a-a31c-70c35dab9738-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44\" (UID: \"209e47ed-5313-484a-a31c-70c35dab9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"
Sep 16 19:03:22.087223 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:22.087203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209e47ed-5313-484a-a31c-70c35dab9738-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44\" (UID: \"209e47ed-5313-484a-a31c-70c35dab9738\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"
Sep 16 19:03:22.194880 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:22.194796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"
Sep 16 19:03:22.491070 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:22.491046    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"]
Sep 16 19:03:22.493051 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:03:22.493021    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209e47ed_5313_484a_a31c_70c35dab9738.slice/crio-9146313266c8969a15ba65ca3b904fa84eba0bb6f27ab5cd16189877c6e8a2a2 WatchSource:0}: Error finding container 9146313266c8969a15ba65ca3b904fa84eba0bb6f27ab5cd16189877c6e8a2a2: Status 404 returned error can't find the container with id 9146313266c8969a15ba65ca3b904fa84eba0bb6f27ab5cd16189877c6e8a2a2
Sep 16 19:03:22.959243 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:22.959206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44" event={"ID":"209e47ed-5313-484a-a31c-70c35dab9738","Type":"ContainerStarted","Data":"a5150d8c654f71e5df9c92b95fe9dd564ed7911e44cb49618fe19ef7b1247e25"}
Sep 16 19:03:22.959751 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:22.959263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44" event={"ID":"209e47ed-5313-484a-a31c-70c35dab9738","Type":"ContainerStarted","Data":"9146313266c8969a15ba65ca3b904fa84eba0bb6f27ab5cd16189877c6e8a2a2"}
Sep 16 19:03:22.980938 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:22.980886    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44" podStartSLOduration=1.980870525 podStartE2EDuration="1.980870525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:22.978208763 +0000 UTC m=+2145.687448155" watchObservedRunningTime="2026-09-16 19:03:22.980870525 +0000 UTC m=+2145.690109915"
Sep 16 19:03:23.965559 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:23.965527    2580 generic.go:356] "Generic (PLEG): container finished" podID="209e47ed-5313-484a-a31c-70c35dab9738" containerID="a5150d8c654f71e5df9c92b95fe9dd564ed7911e44cb49618fe19ef7b1247e25" exitCode=0
Sep 16 19:03:23.966015 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:23.965601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44" event={"ID":"209e47ed-5313-484a-a31c-70c35dab9738","Type":"ContainerDied","Data":"a5150d8c654f71e5df9c92b95fe9dd564ed7911e44cb49618fe19ef7b1247e25"}
Sep 16 19:03:25.101671 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:25.101642    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"
Sep 16 19:03:25.217975 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:25.217946    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209e47ed-5313-484a-a31c-70c35dab9738-maas-api-service-ca\") pod \"209e47ed-5313-484a-a31c-70c35dab9738\" (UID: \"209e47ed-5313-484a-a31c-70c35dab9738\") "
Sep 16 19:03:25.218316 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:25.218292    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209e47ed-5313-484a-a31c-70c35dab9738-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "209e47ed-5313-484a-a31c-70c35dab9738" (UID: "209e47ed-5313-484a-a31c-70c35dab9738"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:25.318787 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:25.318750    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209e47ed-5313-484a-a31c-70c35dab9738-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:03:25.977022 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:25.976994    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"
Sep 16 19:03:25.977022 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:25.977007    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44" event={"ID":"209e47ed-5313-484a-a31c-70c35dab9738","Type":"ContainerDied","Data":"9146313266c8969a15ba65ca3b904fa84eba0bb6f27ab5cd16189877c6e8a2a2"}
Sep 16 19:03:25.977236 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:25.977056    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9146313266c8969a15ba65ca3b904fa84eba0bb6f27ab5cd16189877c6e8a2a2"
Sep 16 19:03:33.168447 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.168415    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"]
Sep 16 19:03:33.168881 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.168656    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf" podUID="f3ee7355-9180-42af-acd6-92a8c524dd5c" containerName="payload-processing" containerID="cri-o://1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012" gracePeriod=30
Sep 16 19:03:33.183669 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.183640    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"]
Sep 16 19:03:33.183858 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.183830    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg" podUID="f0598979-4760-4894-8551-77deae04f845" containerName="payload-pre-processing" containerID="cri-o://590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f" gracePeriod=30
Sep 16 19:03:33.386487 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.386426    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"]
Sep 16 19:03:33.389057 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.389035    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0e536c-v9d44"]
Sep 16 19:03:33.457602 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.457580    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:33.460516 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.460494    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:33.604197 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.604161    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0598979-4760-4894-8551-77deae04f845-plugins-config-volume\") pod \"f0598979-4760-4894-8551-77deae04f845\" (UID: \"f0598979-4760-4894-8551-77deae04f845\") "
Sep 16 19:03:33.604437 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.604251    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww5kn\" (UniqueName: \"kubernetes.io/projected/f3ee7355-9180-42af-acd6-92a8c524dd5c-kube-api-access-ww5kn\") pod \"f3ee7355-9180-42af-acd6-92a8c524dd5c\" (UID: \"f3ee7355-9180-42af-acd6-92a8c524dd5c\") "
Sep 16 19:03:33.604437 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.604280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ee7355-9180-42af-acd6-92a8c524dd5c-plugins-config-volume\") pod \"f3ee7355-9180-42af-acd6-92a8c524dd5c\" (UID: \"f3ee7355-9180-42af-acd6-92a8c524dd5c\") "
Sep 16 19:03:33.604437 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.604373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78hwt\" (UniqueName: \"kubernetes.io/projected/f0598979-4760-4894-8551-77deae04f845-kube-api-access-78hwt\") pod \"f0598979-4760-4894-8551-77deae04f845\" (UID: \"f0598979-4760-4894-8551-77deae04f845\") "
Sep 16 19:03:33.604602 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.604536    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0598979-4760-4894-8551-77deae04f845-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0598979-4760-4894-8551-77deae04f845" (UID: "f0598979-4760-4894-8551-77deae04f845"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:33.604747 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.604726    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ee7355-9180-42af-acd6-92a8c524dd5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3ee7355-9180-42af-acd6-92a8c524dd5c" (UID: "f3ee7355-9180-42af-acd6-92a8c524dd5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:33.604801 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.604730    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0598979-4760-4894-8551-77deae04f845-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:03:33.606456 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.606440    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ee7355-9180-42af-acd6-92a8c524dd5c-kube-api-access-ww5kn" (OuterVolumeSpecName: "kube-api-access-ww5kn") pod "f3ee7355-9180-42af-acd6-92a8c524dd5c" (UID: "f3ee7355-9180-42af-acd6-92a8c524dd5c"). InnerVolumeSpecName "kube-api-access-ww5kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:33.606576 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.606557    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0598979-4760-4894-8551-77deae04f845-kube-api-access-78hwt" (OuterVolumeSpecName: "kube-api-access-78hwt") pod "f0598979-4760-4894-8551-77deae04f845" (UID: "f0598979-4760-4894-8551-77deae04f845"). InnerVolumeSpecName "kube-api-access-78hwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:33.705859 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.705787    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78hwt\" (UniqueName: \"kubernetes.io/projected/f0598979-4760-4894-8551-77deae04f845-kube-api-access-78hwt\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:03:33.705859 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.705816    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5kn\" (UniqueName: \"kubernetes.io/projected/f3ee7355-9180-42af-acd6-92a8c524dd5c-kube-api-access-ww5kn\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:03:33.705859 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.705825    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ee7355-9180-42af-acd6-92a8c524dd5c-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:03:33.784100 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:33.784068    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="209e47ed-5313-484a-a31c-70c35dab9738" path="/var/lib/kubelet/pods/209e47ed-5313-484a-a31c-70c35dab9738/volumes"
Sep 16 19:03:34.021112 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.021077    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0598979-4760-4894-8551-77deae04f845" containerID="590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f" exitCode=0
Sep 16 19:03:34.021302 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.021164    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"
Sep 16 19:03:34.021302 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.021165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg" event={"ID":"f0598979-4760-4894-8551-77deae04f845","Type":"ContainerDied","Data":"590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f"}
Sep 16 19:03:34.021302 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.021280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg" event={"ID":"f0598979-4760-4894-8551-77deae04f845","Type":"ContainerDied","Data":"b4559591fd356ef0792ca15e3703f6ca4352a3019e5e03a3c1914eb8a1861c1a"}
Sep 16 19:03:34.021519 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.021318    2580 scope.go:117] "RemoveContainer" containerID="590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f"
Sep 16 19:03:34.023005 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.022872    2580 generic.go:356] "Generic (PLEG): container finished" podID="f3ee7355-9180-42af-acd6-92a8c524dd5c" containerID="1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012" exitCode=0
Sep 16 19:03:34.023005 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.022909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf" event={"ID":"f3ee7355-9180-42af-acd6-92a8c524dd5c","Type":"ContainerDied","Data":"1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012"}
Sep 16 19:03:34.023005 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.022924    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"
Sep 16 19:03:34.023005 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.022972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf" event={"ID":"f3ee7355-9180-42af-acd6-92a8c524dd5c","Type":"ContainerDied","Data":"608571a7e27894723ed4e6b61176991b948d8972060526be715013a41b1020cd"}
Sep 16 19:03:34.030700 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.030685    2580 scope.go:117] "RemoveContainer" containerID="590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f"
Sep 16 19:03:34.030945 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:03:34.030925    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f\": container with ID starting with 590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f not found: ID does not exist" containerID="590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f"
Sep 16 19:03:34.031004 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.030952    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f"} err="failed to get container status \"590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f\": rpc error: code = NotFound desc = could not find container \"590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f\": container with ID starting with 590cd5fa68c3700c3bab00d6516cabd0c30eb4c5b526c130735e89044bf89e8f not found: ID does not exist"
Sep 16 19:03:34.031004 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.030968    2580 scope.go:117] "RemoveContainer" containerID="1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012"
Sep 16 19:03:34.038319 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.038304    2580 scope.go:117] "RemoveContainer" containerID="1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012"
Sep 16 19:03:34.038580 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:03:34.038562    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012\": container with ID starting with 1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012 not found: ID does not exist" containerID="1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012"
Sep 16 19:03:34.038634 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.038588    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012"} err="failed to get container status \"1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012\": rpc error: code = NotFound desc = could not find container \"1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012\": container with ID starting with 1f3d39f4f05e3fc29ba35588cf818ef52ce4f84e7877609b049077ef712c7012 not found: ID does not exist"
Sep 16 19:03:34.046999 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.046977    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"]
Sep 16 19:03:34.060023 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.060001    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0e536c-948d6f9f5-d2lzf"]
Sep 16 19:03:34.077566 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.077541    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"]
Sep 16 19:03:34.096059 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:34.096029    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0e536c-84fcb48fd-xb2pg"]
Sep 16 19:03:35.783214 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:35.783182    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0598979-4760-4894-8551-77deae04f845" path="/var/lib/kubelet/pods/f0598979-4760-4894-8551-77deae04f845/volumes"
Sep 16 19:03:35.783625 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:35.783565    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3ee7355-9180-42af-acd6-92a8c524dd5c" path="/var/lib/kubelet/pods/f3ee7355-9180-42af-acd6-92a8c524dd5c/volumes"
Sep 16 19:03:37.448953 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:37.448922    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:03:45.745401 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.745364    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"]
Sep 16 19:03:45.745981 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.745961    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ee7355-9180-42af-acd6-92a8c524dd5c" containerName="payload-processing"
Sep 16 19:03:45.746076 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.745982    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ee7355-9180-42af-acd6-92a8c524dd5c" containerName="payload-processing"
Sep 16 19:03:45.746076 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.746000    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0598979-4760-4894-8551-77deae04f845" containerName="payload-pre-processing"
Sep 16 19:03:45.746076 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.746009    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0598979-4760-4894-8551-77deae04f845" containerName="payload-pre-processing"
Sep 16 19:03:45.746076 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.746033    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209e47ed-5313-484a-a31c-70c35dab9738" containerName="revoke-keys"
Sep 16 19:03:45.746076 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.746042    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="209e47ed-5313-484a-a31c-70c35dab9738" containerName="revoke-keys"
Sep 16 19:03:45.746324 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.746143    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="209e47ed-5313-484a-a31c-70c35dab9738" containerName="revoke-keys"
Sep 16 19:03:45.746324 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.746156    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3ee7355-9180-42af-acd6-92a8c524dd5c" containerName="payload-processing"
Sep 16 19:03:45.746324 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.746171    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0598979-4760-4894-8551-77deae04f845" containerName="payload-pre-processing"
Sep 16 19:03:45.749290 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.749268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"
Sep 16 19:03:45.751768 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.751750    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:03:45.751872 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.751748    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 19:03:45.757296 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.757272    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"]
Sep 16 19:03:45.810457 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.810426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dacb9e6-43a4-4e86-89e1-26d230861602-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj\" (UID: \"0dacb9e6-43a4-4e86-89e1-26d230861602\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"
Sep 16 19:03:45.911166 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.911127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dacb9e6-43a4-4e86-89e1-26d230861602-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj\" (UID: \"0dacb9e6-43a4-4e86-89e1-26d230861602\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"
Sep 16 19:03:45.911792 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:45.911772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dacb9e6-43a4-4e86-89e1-26d230861602-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj\" (UID: \"0dacb9e6-43a4-4e86-89e1-26d230861602\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"
Sep 16 19:03:46.060882 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:46.060848    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"
Sep 16 19:03:46.348747 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:46.348716    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"]
Sep 16 19:03:46.350824 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:03:46.350798    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dacb9e6_43a4_4e86_89e1_26d230861602.slice/crio-0bf3df69715f491b5e4bfe06f67c085294c995e21d84276c5a60228db03f1bcb WatchSource:0}: Error finding container 0bf3df69715f491b5e4bfe06f67c085294c995e21d84276c5a60228db03f1bcb: Status 404 returned error can't find the container with id 0bf3df69715f491b5e4bfe06f67c085294c995e21d84276c5a60228db03f1bcb
Sep 16 19:03:47.095839 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:47.095809    2580 generic.go:356] "Generic (PLEG): container finished" podID="0dacb9e6-43a4-4e86-89e1-26d230861602" containerID="db07073a03a0b7e518a4b0a9723f22c38511e6fa9b0298a4264a7225a64d8ac1" exitCode=0
Sep 16 19:03:47.096173 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:47.095892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj" event={"ID":"0dacb9e6-43a4-4e86-89e1-26d230861602","Type":"ContainerDied","Data":"db07073a03a0b7e518a4b0a9723f22c38511e6fa9b0298a4264a7225a64d8ac1"}
Sep 16 19:03:47.096173 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:47.095943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj" event={"ID":"0dacb9e6-43a4-4e86-89e1-26d230861602","Type":"ContainerStarted","Data":"0bf3df69715f491b5e4bfe06f67c085294c995e21d84276c5a60228db03f1bcb"}
Sep 16 19:03:48.238713 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:48.238689    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"
Sep 16 19:03:48.329594 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:48.329566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dacb9e6-43a4-4e86-89e1-26d230861602-maas-api-service-ca\") pod \"0dacb9e6-43a4-4e86-89e1-26d230861602\" (UID: \"0dacb9e6-43a4-4e86-89e1-26d230861602\") "
Sep 16 19:03:48.329940 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:48.329918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dacb9e6-43a4-4e86-89e1-26d230861602-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dacb9e6-43a4-4e86-89e1-26d230861602" (UID: "0dacb9e6-43a4-4e86-89e1-26d230861602"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:48.431122 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:48.431044    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dacb9e6-43a4-4e86-89e1-26d230861602-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:03:49.106839 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:49.106808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj" event={"ID":"0dacb9e6-43a4-4e86-89e1-26d230861602","Type":"ContainerDied","Data":"0bf3df69715f491b5e4bfe06f67c085294c995e21d84276c5a60228db03f1bcb"}
Sep 16 19:03:49.107088 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:49.106850    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bf3df69715f491b5e4bfe06f67c085294c995e21d84276c5a60228db03f1bcb"
Sep 16 19:03:49.107088 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:03:49.106813    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"
Sep 16 19:04:02.449290 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:02.449253    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:04:08.478550 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:08.478520    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"]
Sep 16 19:04:08.487671 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:08.487646    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a532e-zzxjj"]
Sep 16 19:04:09.790646 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:09.790608    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dacb9e6-43a4-4e86-89e1-26d230861602" path="/var/lib/kubelet/pods/0dacb9e6-43a4-4e86-89e1-26d230861602/volumes"
Sep 16 19:04:13.035497 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:13.035464    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:04:59.542921 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:59.542888    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"]
Sep 16 19:04:59.543519 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:59.543095    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy" containerID="cri-o://33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4" gracePeriod=30
Sep 16 19:04:59.950483 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:59.950401    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"]
Sep 16 19:04:59.950785 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:59.950773    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dacb9e6-43a4-4e86-89e1-26d230861602" containerName="revoke-keys"
Sep 16 19:04:59.950829 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:59.950787    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dacb9e6-43a4-4e86-89e1-26d230861602" containerName="revoke-keys"
Sep 16 19:04:59.950864 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:59.950852    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dacb9e6-43a4-4e86-89e1-26d230861602" containerName="revoke-keys"
Sep 16 19:04:59.954042 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:59.954020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"
Sep 16 19:04:59.956983 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:59.956963    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 19:04:59.957097 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:59.956965    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:04:59.966160 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:04:59.966108    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"]
Sep 16 19:05:00.085882 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:00.085836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv\" (UID: \"fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"
Sep 16 19:05:00.186606 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:00.186558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv\" (UID: \"fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"
Sep 16 19:05:00.187179 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:00.187159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv\" (UID: \"fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"
Sep 16 19:05:00.265162 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:00.265080    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"
Sep 16 19:05:00.538252 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:00.538226    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"]
Sep 16 19:05:00.540303 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:05:00.540276    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfb07b2_d385_4c4a_9c8c_63a7f2c36ab2.slice/crio-6615e2cc830e2e10bf502579b5da95cd922a6218f6b5e4410892ac9987d48337 WatchSource:0}: Error finding container 6615e2cc830e2e10bf502579b5da95cd922a6218f6b5e4410892ac9987d48337: Status 404 returned error can't find the container with id 6615e2cc830e2e10bf502579b5da95cd922a6218f6b5e4410892ac9987d48337
Sep 16 19:05:01.472021 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:01.471968    2580 generic.go:356] "Generic (PLEG): container finished" podID="fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2" containerID="0f130fe42b8656a1a7e49f5dbc7fa028444abc7cfe4b29a914b5822a37b14575" exitCode=0
Sep 16 19:05:01.472476 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:01.472172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv" event={"ID":"fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2","Type":"ContainerDied","Data":"0f130fe42b8656a1a7e49f5dbc7fa028444abc7cfe4b29a914b5822a37b14575"}
Sep 16 19:05:01.472476 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:01.472237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv" event={"ID":"fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2","Type":"ContainerStarted","Data":"6615e2cc830e2e10bf502579b5da95cd922a6218f6b5e4410892ac9987d48337"}
Sep 16 19:05:02.013528 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:02.013487    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 19:05:02.013712 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:02.013550    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:05:02.601923 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:02.601894    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"
Sep 16 19:05:02.711082 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:02.711051    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2-maas-api-service-ca\") pod \"fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2\" (UID: \"fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2\") "
Sep 16 19:05:02.711490 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:02.711469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2" (UID: "fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:02.812773 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:02.812738    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:03.483801 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:03.483774    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"
Sep 16 19:05:03.483983 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:03.483807    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv" event={"ID":"fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2","Type":"ContainerDied","Data":"6615e2cc830e2e10bf502579b5da95cd922a6218f6b5e4410892ac9987d48337"}
Sep 16 19:05:03.483983 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:03.483849    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6615e2cc830e2e10bf502579b5da95cd922a6218f6b5e4410892ac9987d48337"
Sep 16 19:05:04.807522 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.807449    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:05:04.935227 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935191    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-istio-token\") pod \"ee750069-54c9-439b-84ee-0c96942cd03e\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") "
Sep 16 19:05:04.935438 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935285    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-envoy\") pod \"ee750069-54c9-439b-84ee-0c96942cd03e\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") "
Sep 16 19:05:04.935438 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-socket\") pod \"ee750069-54c9-439b-84ee-0c96942cd03e\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") "
Sep 16 19:05:04.935438 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935423    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-data\") pod \"ee750069-54c9-439b-84ee-0c96942cd03e\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") "
Sep 16 19:05:04.935591 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee750069-54c9-439b-84ee-0c96942cd03e-istiod-ca-cert\") pod \"ee750069-54c9-439b-84ee-0c96942cd03e\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") "
Sep 16 19:05:04.935591 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb8h4\" (UniqueName: \"kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-kube-api-access-qb8h4\") pod \"ee750069-54c9-439b-84ee-0c96942cd03e\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") "
Sep 16 19:05:04.935591 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935552    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-certs\") pod \"ee750069-54c9-439b-84ee-0c96942cd03e\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") "
Sep 16 19:05:04.935757 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935592    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-credential-socket\") pod \"ee750069-54c9-439b-84ee-0c96942cd03e\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") "
Sep 16 19:05:04.935757 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee750069-54c9-439b-84ee-0c96942cd03e-istio-podinfo\") pod \"ee750069-54c9-439b-84ee-0c96942cd03e\" (UID: \"ee750069-54c9-439b-84ee-0c96942cd03e\") "
Sep 16 19:05:04.935757 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935629    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee750069-54c9-439b-84ee-0c96942cd03e" (UID: "ee750069-54c9-439b-84ee-0c96942cd03e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:04.935757 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935664    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee750069-54c9-439b-84ee-0c96942cd03e" (UID: "ee750069-54c9-439b-84ee-0c96942cd03e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:04.935973 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee750069-54c9-439b-84ee-0c96942cd03e" (UID: "ee750069-54c9-439b-84ee-0c96942cd03e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:04.935973 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.935886    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee750069-54c9-439b-84ee-0c96942cd03e" (UID: "ee750069-54c9-439b-84ee-0c96942cd03e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:04.936077 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.936017    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:04.936077 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.936037    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-data\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:04.936077 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.936052    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-workload-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:04.936077 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.936065    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-credential-socket\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:04.936454 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.936431    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee750069-54c9-439b-84ee-0c96942cd03e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee750069-54c9-439b-84ee-0c96942cd03e" (UID: "ee750069-54c9-439b-84ee-0c96942cd03e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:04.937985 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.937953    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee750069-54c9-439b-84ee-0c96942cd03e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee750069-54c9-439b-84ee-0c96942cd03e" (UID: "ee750069-54c9-439b-84ee-0c96942cd03e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:05:04.938098 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.938004    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-kube-api-access-qb8h4" (OuterVolumeSpecName: "kube-api-access-qb8h4") pod "ee750069-54c9-439b-84ee-0c96942cd03e" (UID: "ee750069-54c9-439b-84ee-0c96942cd03e"). InnerVolumeSpecName "kube-api-access-qb8h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:04.938098 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.938020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee750069-54c9-439b-84ee-0c96942cd03e" (UID: "ee750069-54c9-439b-84ee-0c96942cd03e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:04.938181 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:04.938091    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee750069-54c9-439b-84ee-0c96942cd03e" (UID: "ee750069-54c9-439b-84ee-0c96942cd03e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:05.036708 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.036674    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee750069-54c9-439b-84ee-0c96942cd03e-istio-podinfo\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:05.036708 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.036699    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-istio-token\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:05.036708 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.036708    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee750069-54c9-439b-84ee-0c96942cd03e-istio-envoy\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:05.036708 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.036716    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee750069-54c9-439b-84ee-0c96942cd03e-istiod-ca-cert\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:05.036944 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.036724    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb8h4\" (UniqueName: \"kubernetes.io/projected/ee750069-54c9-439b-84ee-0c96942cd03e-kube-api-access-qb8h4\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:05.496278 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.496238    2580 generic.go:356] "Generic (PLEG): container finished" podID="ee750069-54c9-439b-84ee-0c96942cd03e" containerID="33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4" exitCode=0
Sep 16 19:05:05.496510 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.496299    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"
Sep 16 19:05:05.496510 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.496324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" event={"ID":"ee750069-54c9-439b-84ee-0c96942cd03e","Type":"ContainerDied","Data":"33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4"}
Sep 16 19:05:05.496510 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.496449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c" event={"ID":"ee750069-54c9-439b-84ee-0c96942cd03e","Type":"ContainerDied","Data":"d887a0219f889b431866bd0fafc36bf337975f7298a20a38c7e9e2e25bbf943d"}
Sep 16 19:05:05.496510 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.496480    2580 scope.go:117] "RemoveContainer" containerID="33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4"
Sep 16 19:05:05.506823 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.506801    2580 scope.go:117] "RemoveContainer" containerID="f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e"
Sep 16 19:05:05.515139 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.515121    2580 scope.go:117] "RemoveContainer" containerID="33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4"
Sep 16 19:05:05.515430 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:05:05.515409    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4\": container with ID starting with 33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4 not found: ID does not exist" containerID="33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4"
Sep 16 19:05:05.515490 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.515440    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4"} err="failed to get container status \"33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4\": rpc error: code = NotFound desc = could not find container \"33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4\": container with ID starting with 33991d10336a771102a78b599f2fddce097c995d92c8bcaf57fd2d19269a8ae4 not found: ID does not exist"
Sep 16 19:05:05.515490 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.515460    2580 scope.go:117] "RemoveContainer" containerID="f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e"
Sep 16 19:05:05.515765 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:05:05.515747    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e\": container with ID starting with f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e not found: ID does not exist" containerID="f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e"
Sep 16 19:05:05.515817 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.515773    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e"} err="failed to get container status \"f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e\": rpc error: code = NotFound desc = could not find container \"f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e\": container with ID starting with f23d6c8bd005485ea1da3c33c667bc0cbd3fe6c91b837a509b865d6b4d721f1e not found: ID does not exist"
Sep 16 19:05:05.523084 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.523062    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"]
Sep 16 19:05:05.526341 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.526302    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a532e-openshift-default-5bbd57667d-scf2c"]
Sep 16 19:05:05.784177 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:05.784147    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" path="/var/lib/kubelet/pods/ee750069-54c9-439b-84ee-0c96942cd03e/volumes"
Sep 16 19:05:10.391422 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.391385    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"]
Sep 16 19:05:10.391909 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.391695    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp" podUID="e12a3805-161e-4080-a16d-17913321de50" containerName="payload-processing" containerID="cri-o://234a7a35466fcc7eddd39c4dcac1373b5ad2fdd244ae7f1b8eef22f3546b58d0" gracePeriod=30
Sep 16 19:05:10.395480 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.395451    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"]
Sep 16 19:05:10.396032 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.395983    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9" podUID="bed4bd4e-5a32-4652-a0a3-941f7824239f" containerName="payload-pre-processing" containerID="cri-o://7352dfefc650292c31ddd318b5d57dc8c7c0a55082d1a22d40286c40452a91b3" gracePeriod=30
Sep 16 19:05:10.527199 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.527125    2580 generic.go:356] "Generic (PLEG): container finished" podID="e12a3805-161e-4080-a16d-17913321de50" containerID="234a7a35466fcc7eddd39c4dcac1373b5ad2fdd244ae7f1b8eef22f3546b58d0" exitCode=0
Sep 16 19:05:10.527394 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.527204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp" event={"ID":"e12a3805-161e-4080-a16d-17913321de50","Type":"ContainerDied","Data":"234a7a35466fcc7eddd39c4dcac1373b5ad2fdd244ae7f1b8eef22f3546b58d0"}
Sep 16 19:05:10.529087 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.529009    2580 generic.go:356] "Generic (PLEG): container finished" podID="bed4bd4e-5a32-4652-a0a3-941f7824239f" containerID="7352dfefc650292c31ddd318b5d57dc8c7c0a55082d1a22d40286c40452a91b3" exitCode=0
Sep 16 19:05:10.529087 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.529042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9" event={"ID":"bed4bd4e-5a32-4652-a0a3-941f7824239f","Type":"ContainerDied","Data":"7352dfefc650292c31ddd318b5d57dc8c7c0a55082d1a22d40286c40452a91b3"}
Sep 16 19:05:10.670257 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.670229    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:05:10.673433 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.673412    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:05:10.772516 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.772483    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"]
Sep 16 19:05:10.782081 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.782053    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a0f152-mpbtv"]
Sep 16 19:05:10.785434 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.785415    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpmfs\" (UniqueName: \"kubernetes.io/projected/e12a3805-161e-4080-a16d-17913321de50-kube-api-access-tpmfs\") pod \"e12a3805-161e-4080-a16d-17913321de50\" (UID: \"e12a3805-161e-4080-a16d-17913321de50\") "
Sep 16 19:05:10.785543 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.785465    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed4bd4e-5a32-4652-a0a3-941f7824239f-plugins-config-volume\") pod \"bed4bd4e-5a32-4652-a0a3-941f7824239f\" (UID: \"bed4bd4e-5a32-4652-a0a3-941f7824239f\") "
Sep 16 19:05:10.785543 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.785484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcckb\" (UniqueName: \"kubernetes.io/projected/bed4bd4e-5a32-4652-a0a3-941f7824239f-kube-api-access-fcckb\") pod \"bed4bd4e-5a32-4652-a0a3-941f7824239f\" (UID: \"bed4bd4e-5a32-4652-a0a3-941f7824239f\") "
Sep 16 19:05:10.785543 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.785506    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12a3805-161e-4080-a16d-17913321de50-plugins-config-volume\") pod \"e12a3805-161e-4080-a16d-17913321de50\" (UID: \"e12a3805-161e-4080-a16d-17913321de50\") "
Sep 16 19:05:10.785889 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.785857    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed4bd4e-5a32-4652-a0a3-941f7824239f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bed4bd4e-5a32-4652-a0a3-941f7824239f" (UID: "bed4bd4e-5a32-4652-a0a3-941f7824239f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:10.785994 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.785887    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12a3805-161e-4080-a16d-17913321de50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e12a3805-161e-4080-a16d-17913321de50" (UID: "e12a3805-161e-4080-a16d-17913321de50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:10.787713 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.787689    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed4bd4e-5a32-4652-a0a3-941f7824239f-kube-api-access-fcckb" (OuterVolumeSpecName: "kube-api-access-fcckb") pod "bed4bd4e-5a32-4652-a0a3-941f7824239f" (UID: "bed4bd4e-5a32-4652-a0a3-941f7824239f"). InnerVolumeSpecName "kube-api-access-fcckb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:10.787713 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.787706    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12a3805-161e-4080-a16d-17913321de50-kube-api-access-tpmfs" (OuterVolumeSpecName: "kube-api-access-tpmfs") pod "e12a3805-161e-4080-a16d-17913321de50" (UID: "e12a3805-161e-4080-a16d-17913321de50"). InnerVolumeSpecName "kube-api-access-tpmfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:10.886986 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.886954    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpmfs\" (UniqueName: \"kubernetes.io/projected/e12a3805-161e-4080-a16d-17913321de50-kube-api-access-tpmfs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:10.886986 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.886981    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed4bd4e-5a32-4652-a0a3-941f7824239f-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:10.886986 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.886991    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcckb\" (UniqueName: \"kubernetes.io/projected/bed4bd4e-5a32-4652-a0a3-941f7824239f-kube-api-access-fcckb\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:10.887207 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:10.886999    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12a3805-161e-4080-a16d-17913321de50-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:11.535230 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.535196    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"
Sep 16 19:05:11.535754 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.535227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9" event={"ID":"bed4bd4e-5a32-4652-a0a3-941f7824239f","Type":"ContainerDied","Data":"0d18b9c455314a7418b3acb5a256979672c64606623561f389320841619f3514"}
Sep 16 19:05:11.535754 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.535293    2580 scope.go:117] "RemoveContainer" containerID="7352dfefc650292c31ddd318b5d57dc8c7c0a55082d1a22d40286c40452a91b3"
Sep 16 19:05:11.536874 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.536848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp" event={"ID":"e12a3805-161e-4080-a16d-17913321de50","Type":"ContainerDied","Data":"9c8b9e956e769b25ab5e79257b07f22bfd2b9bc7bea16272f16a077e82e4ebb6"}
Sep 16 19:05:11.536874 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.536871    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"
Sep 16 19:05:11.545150 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.545134    2580 scope.go:117] "RemoveContainer" containerID="234a7a35466fcc7eddd39c4dcac1373b5ad2fdd244ae7f1b8eef22f3546b58d0"
Sep 16 19:05:11.563140 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.563119    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"]
Sep 16 19:05:11.566837 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.566816    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a0f152-5cd548dbb6-r6xk9"]
Sep 16 19:05:11.582306 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.582287    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"]
Sep 16 19:05:11.588359 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.588318    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a0f152-5fd487bdfc-jfgxp"]
Sep 16 19:05:11.783496 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.783466    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bed4bd4e-5a32-4652-a0a3-941f7824239f" path="/var/lib/kubelet/pods/bed4bd4e-5a32-4652-a0a3-941f7824239f/volumes"
Sep 16 19:05:11.783845 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.783833    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e12a3805-161e-4080-a16d-17913321de50" path="/var/lib/kubelet/pods/e12a3805-161e-4080-a16d-17913321de50/volumes"
Sep 16 19:05:11.784126 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:11.784114    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2" path="/var/lib/kubelet/pods/fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2/volumes"
Sep 16 19:05:13.944817 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:13.944784    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:05:19.916376 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.916313    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"]
Sep 16 19:05:19.916850 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.916831    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2" containerName="revoke-keys"
Sep 16 19:05:19.916896 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.916853    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2" containerName="revoke-keys"
Sep 16 19:05:19.916896 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.916876    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy"
Sep 16 19:05:19.916896 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.916883    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy"
Sep 16 19:05:19.916993 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.916897    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12a3805-161e-4080-a16d-17913321de50" containerName="payload-processing"
Sep 16 19:05:19.916993 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.916905    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12a3805-161e-4080-a16d-17913321de50" containerName="payload-processing"
Sep 16 19:05:19.916993 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.916916    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed4bd4e-5a32-4652-a0a3-941f7824239f" containerName="payload-pre-processing"
Sep 16 19:05:19.916993 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.916924    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed4bd4e-5a32-4652-a0a3-941f7824239f" containerName="payload-pre-processing"
Sep 16 19:05:19.917109 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.917007    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bed4bd4e-5a32-4652-a0a3-941f7824239f" containerName="payload-pre-processing"
Sep 16 19:05:19.917109 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.917016    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12a3805-161e-4080-a16d-17913321de50" containerName="payload-processing"
Sep 16 19:05:19.917109 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.917027    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy"
Sep 16 19:05:19.917109 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.917035    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy"
Sep 16 19:05:19.917109 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.917048    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcfb07b2-d385-4c4a-9c8c-63a7f2c36ab2" containerName="revoke-keys"
Sep 16 19:05:19.920594 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.920577    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"
Sep 16 19:05:19.932386 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.932361    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:05:19.932521 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.932370    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 19:05:19.933548 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.933526    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"]
Sep 16 19:05:19.973069 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:19.973039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f27daa-c340-442e-aec6-0739b5bd9ee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc\" (UID: \"a9f27daa-c340-442e-aec6-0739b5bd9ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"
Sep 16 19:05:20.075141 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:20.075096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f27daa-c340-442e-aec6-0739b5bd9ee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc\" (UID: \"a9f27daa-c340-442e-aec6-0739b5bd9ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"
Sep 16 19:05:20.075785 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:20.075760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f27daa-c340-442e-aec6-0739b5bd9ee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc\" (UID: \"a9f27daa-c340-442e-aec6-0739b5bd9ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"
Sep 16 19:05:20.230585 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:20.230483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"
Sep 16 19:05:20.511418 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:20.511340    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"]
Sep 16 19:05:20.515069 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:05:20.515039    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f27daa_c340_442e_aec6_0739b5bd9ee1.slice/crio-aa950195eb7155816ae4a8916f8a90620041b735fd73c5532add8058175519fa WatchSource:0}: Error finding container aa950195eb7155816ae4a8916f8a90620041b735fd73c5532add8058175519fa: Status 404 returned error can't find the container with id aa950195eb7155816ae4a8916f8a90620041b735fd73c5532add8058175519fa
Sep 16 19:05:20.590069 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:20.590029    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc" event={"ID":"a9f27daa-c340-442e-aec6-0739b5bd9ee1","Type":"ContainerStarted","Data":"aa950195eb7155816ae4a8916f8a90620041b735fd73c5532add8058175519fa"}
Sep 16 19:05:21.597717 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:21.597681    2580 generic.go:356] "Generic (PLEG): container finished" podID="a9f27daa-c340-442e-aec6-0739b5bd9ee1" containerID="88116a47180f381efa7f184f3e74eb6cfc20e3e38fbb91dbdbbfc347146367c0" exitCode=0
Sep 16 19:05:21.598121 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:21.597776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc" event={"ID":"a9f27daa-c340-442e-aec6-0739b5bd9ee1","Type":"ContainerDied","Data":"88116a47180f381efa7f184f3e74eb6cfc20e3e38fbb91dbdbbfc347146367c0"}
Sep 16 19:05:22.737893 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:22.737869    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"
Sep 16 19:05:22.802720 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:22.802692    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f27daa-c340-442e-aec6-0739b5bd9ee1-maas-api-service-ca\") pod \"a9f27daa-c340-442e-aec6-0739b5bd9ee1\" (UID: \"a9f27daa-c340-442e-aec6-0739b5bd9ee1\") "
Sep 16 19:05:22.803094 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:22.803076    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f27daa-c340-442e-aec6-0739b5bd9ee1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9f27daa-c340-442e-aec6-0739b5bd9ee1" (UID: "a9f27daa-c340-442e-aec6-0739b5bd9ee1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:22.903616 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:22.903538    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f27daa-c340-442e-aec6-0739b5bd9ee1-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:23.609506 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:23.609472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc" event={"ID":"a9f27daa-c340-442e-aec6-0739b5bd9ee1","Type":"ContainerDied","Data":"aa950195eb7155816ae4a8916f8a90620041b735fd73c5532add8058175519fa"}
Sep 16 19:05:23.609506 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:23.609509    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"
Sep 16 19:05:23.609722 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:23.609521    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa950195eb7155816ae4a8916f8a90620041b735fd73c5532add8058175519fa"
Sep 16 19:05:33.496879 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:33.496839    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"]
Sep 16 19:05:33.500141 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:33.500114    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f9c9b8-r4qxc"]
Sep 16 19:05:33.783200 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:33.783166    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9f27daa-c340-442e-aec6-0739b5bd9ee1" path="/var/lib/kubelet/pods/a9f27daa-c340-442e-aec6-0739b5bd9ee1/volumes"
Sep 16 19:05:35.927365 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:35.927319    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:05:43.400419 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.400382    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"]
Sep 16 19:05:43.400998 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.400953    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f27daa-c340-442e-aec6-0739b5bd9ee1" containerName="revoke-keys"
Sep 16 19:05:43.401101 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.400999    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f27daa-c340-442e-aec6-0739b5bd9ee1" containerName="revoke-keys"
Sep 16 19:05:43.401101 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.401063    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy"
Sep 16 19:05:43.401101 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.401073    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee750069-54c9-439b-84ee-0c96942cd03e" containerName="istio-proxy"
Sep 16 19:05:43.401252 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.401159    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f27daa-c340-442e-aec6-0739b5bd9ee1" containerName="revoke-keys"
Sep 16 19:05:43.405655 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.405632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"
Sep 16 19:05:43.416555 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.416534    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:05:43.416670 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.416623    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sdhrk\""
Sep 16 19:05:43.436446 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.436413    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"]
Sep 16 19:05:43.493465 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.493425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29af6dd-da7a-49e4-addd-35931065db84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2\" (UID: \"a29af6dd-da7a-49e4-addd-35931065db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"
Sep 16 19:05:43.593949 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.593911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29af6dd-da7a-49e4-addd-35931065db84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2\" (UID: \"a29af6dd-da7a-49e4-addd-35931065db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"
Sep 16 19:05:43.594563 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.594541    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29af6dd-da7a-49e4-addd-35931065db84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2\" (UID: \"a29af6dd-da7a-49e4-addd-35931065db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"
Sep 16 19:05:43.715502 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:43.715421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"
Sep 16 19:05:44.003529 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:44.003501    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"]
Sep 16 19:05:44.005424 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:05:44.005398    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29af6dd_da7a_49e4_addd_35931065db84.slice/crio-08cd19c2786c67093e6d8de6667994f260e1e9d42d959b07403bb4dc3b22fdb4 WatchSource:0}: Error finding container 08cd19c2786c67093e6d8de6667994f260e1e9d42d959b07403bb4dc3b22fdb4: Status 404 returned error can't find the container with id 08cd19c2786c67093e6d8de6667994f260e1e9d42d959b07403bb4dc3b22fdb4
Sep 16 19:05:44.716380 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:44.716347    2580 generic.go:356] "Generic (PLEG): container finished" podID="a29af6dd-da7a-49e4-addd-35931065db84" containerID="f06c16215cac4594cfb7bf8b03866da47f2a18407b8b6ba2311e1c60228c7fc8" exitCode=0
Sep 16 19:05:44.716840 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:44.716421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2" event={"ID":"a29af6dd-da7a-49e4-addd-35931065db84","Type":"ContainerDied","Data":"f06c16215cac4594cfb7bf8b03866da47f2a18407b8b6ba2311e1c60228c7fc8"}
Sep 16 19:05:44.716840 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:44.716513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2" event={"ID":"a29af6dd-da7a-49e4-addd-35931065db84","Type":"ContainerStarted","Data":"08cd19c2786c67093e6d8de6667994f260e1e9d42d959b07403bb4dc3b22fdb4"}
Sep 16 19:05:45.846279 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:45.846256    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"
Sep 16 19:05:46.017429 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:46.017398    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29af6dd-da7a-49e4-addd-35931065db84-maas-api-service-ca\") pod \"a29af6dd-da7a-49e4-addd-35931065db84\" (UID: \"a29af6dd-da7a-49e4-addd-35931065db84\") "
Sep 16 19:05:46.017766 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:46.017744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29af6dd-da7a-49e4-addd-35931065db84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a29af6dd-da7a-49e4-addd-35931065db84" (UID: "a29af6dd-da7a-49e4-addd-35931065db84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:46.118921 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:46.118891    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29af6dd-da7a-49e4-addd-35931065db84-maas-api-service-ca\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:46.727571 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:46.727543    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"
Sep 16 19:05:46.727749 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:46.727585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2" event={"ID":"a29af6dd-da7a-49e4-addd-35931065db84","Type":"ContainerDied","Data":"08cd19c2786c67093e6d8de6667994f260e1e9d42d959b07403bb4dc3b22fdb4"}
Sep 16 19:05:46.727749 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:46.727644    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08cd19c2786c67093e6d8de6667994f260e1e9d42d959b07403bb4dc3b22fdb4"
Sep 16 19:05:55.071913 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.071877    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm"]
Sep 16 19:05:55.076921 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.076889    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e65cc4-29826420-l8fpm"]
Sep 16 19:05:55.099399 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.099372    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"]
Sep 16 19:05:55.099642 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.099616    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk" podUID="bce64fa3-476d-42a0-8b1f-997cdb57da4e" containerName="payload-processing" containerID="cri-o://cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0" gracePeriod=30
Sep 16 19:05:55.112406 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.112379    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"]
Sep 16 19:05:55.112774 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.112749    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd" podUID="8a6375c0-59f8-4f29-864f-ceeaf950042f" containerName="payload-pre-processing" containerID="cri-o://18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1" gracePeriod=30
Sep 16 19:05:55.370112 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.370086    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 19:05:55.373289 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.373265    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"]
Sep 16 19:05:55.377780 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.377760    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e65cc4-ds9p2"]
Sep 16 19:05:55.400279 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.400253    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 19:05:55.400601 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.400583    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce64fa3-476d-42a0-8b1f-997cdb57da4e-plugins-config-volume\") pod \"bce64fa3-476d-42a0-8b1f-997cdb57da4e\" (UID: \"bce64fa3-476d-42a0-8b1f-997cdb57da4e\") "
Sep 16 19:05:55.400684 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.400671    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69vvj\" (UniqueName: \"kubernetes.io/projected/bce64fa3-476d-42a0-8b1f-997cdb57da4e-kube-api-access-69vvj\") pod \"bce64fa3-476d-42a0-8b1f-997cdb57da4e\" (UID: \"bce64fa3-476d-42a0-8b1f-997cdb57da4e\") "
Sep 16 19:05:55.401040 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.401015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce64fa3-476d-42a0-8b1f-997cdb57da4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bce64fa3-476d-42a0-8b1f-997cdb57da4e" (UID: "bce64fa3-476d-42a0-8b1f-997cdb57da4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:55.402725 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.402702    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce64fa3-476d-42a0-8b1f-997cdb57da4e-kube-api-access-69vvj" (OuterVolumeSpecName: "kube-api-access-69vvj") pod "bce64fa3-476d-42a0-8b1f-997cdb57da4e" (UID: "bce64fa3-476d-42a0-8b1f-997cdb57da4e"). InnerVolumeSpecName "kube-api-access-69vvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:55.501372 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.501308    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7sdv\" (UniqueName: \"kubernetes.io/projected/8a6375c0-59f8-4f29-864f-ceeaf950042f-kube-api-access-f7sdv\") pod \"8a6375c0-59f8-4f29-864f-ceeaf950042f\" (UID: \"8a6375c0-59f8-4f29-864f-ceeaf950042f\") "
Sep 16 19:05:55.501523 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.501416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6375c0-59f8-4f29-864f-ceeaf950042f-plugins-config-volume\") pod \"8a6375c0-59f8-4f29-864f-ceeaf950042f\" (UID: \"8a6375c0-59f8-4f29-864f-ceeaf950042f\") "
Sep 16 19:05:55.501604 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.501591    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce64fa3-476d-42a0-8b1f-997cdb57da4e-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:55.501640 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.501607    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69vvj\" (UniqueName: \"kubernetes.io/projected/bce64fa3-476d-42a0-8b1f-997cdb57da4e-kube-api-access-69vvj\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:55.501777 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.501754    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6375c0-59f8-4f29-864f-ceeaf950042f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a6375c0-59f8-4f29-864f-ceeaf950042f" (UID: "8a6375c0-59f8-4f29-864f-ceeaf950042f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:55.503456 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.503432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6375c0-59f8-4f29-864f-ceeaf950042f-kube-api-access-f7sdv" (OuterVolumeSpecName: "kube-api-access-f7sdv") pod "8a6375c0-59f8-4f29-864f-ceeaf950042f" (UID: "8a6375c0-59f8-4f29-864f-ceeaf950042f"). InnerVolumeSpecName "kube-api-access-f7sdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:55.602303 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.602223    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7sdv\" (UniqueName: \"kubernetes.io/projected/8a6375c0-59f8-4f29-864f-ceeaf950042f-kube-api-access-f7sdv\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:55.602303 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.602250    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6375c0-59f8-4f29-864f-ceeaf950042f-plugins-config-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:05:55.774030 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.773996    2580 generic.go:356] "Generic (PLEG): container finished" podID="bce64fa3-476d-42a0-8b1f-997cdb57da4e" containerID="cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0" exitCode=0
Sep 16 19:05:55.774190 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.774044    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"
Sep 16 19:05:55.774190 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.774082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk" event={"ID":"bce64fa3-476d-42a0-8b1f-997cdb57da4e","Type":"ContainerDied","Data":"cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0"}
Sep 16 19:05:55.774190 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.774136    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk" event={"ID":"bce64fa3-476d-42a0-8b1f-997cdb57da4e","Type":"ContainerDied","Data":"ec5f1f97b6ae569054e3dfbb57f0bb2f5504eed8e8d052ed85474ea8da787352"}
Sep 16 19:05:55.774190 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.774162    2580 scope.go:117] "RemoveContainer" containerID="cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0"
Sep 16 19:05:55.783640 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.783613    2580 generic.go:356] "Generic (PLEG): container finished" podID="8a6375c0-59f8-4f29-864f-ceeaf950042f" containerID="18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1" exitCode=0
Sep 16 19:05:55.783830 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.783812    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"
Sep 16 19:05:55.788419 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.788388    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a29af6dd-da7a-49e4-addd-35931065db84" path="/var/lib/kubelet/pods/a29af6dd-da7a-49e4-addd-35931065db84/volumes"
Sep 16 19:05:55.788802 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.788785    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c54b1bc4-2efb-40c3-bbb9-6e8f4af3da0d" path="/var/lib/kubelet/pods/c54b1bc4-2efb-40c3-bbb9-6e8f4af3da0d/volumes"
Sep 16 19:05:55.789318 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.789293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd" event={"ID":"8a6375c0-59f8-4f29-864f-ceeaf950042f","Type":"ContainerDied","Data":"18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1"}
Sep 16 19:05:55.789443 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.789371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd" event={"ID":"8a6375c0-59f8-4f29-864f-ceeaf950042f","Type":"ContainerDied","Data":"3d5c71bf250052d78e6984f593ab141ffc4df78c91a46a8c401601b5d4f2be4c"}
Sep 16 19:05:55.793165 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.793148    2580 scope.go:117] "RemoveContainer" containerID="cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0"
Sep 16 19:05:55.793470 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:05:55.793447    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0\": container with ID starting with cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0 not found: ID does not exist" containerID="cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0"
Sep 16 19:05:55.793570 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.793476    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0"} err="failed to get container status \"cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0\": rpc error: code = NotFound desc = could not find container \"cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0\": container with ID starting with cd7576ffb3a8275320f8f6f928a991c73a53af3718661981c24329bff6aa5fe0 not found: ID does not exist"
Sep 16 19:05:55.793570 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.793490    2580 scope.go:117] "RemoveContainer" containerID="18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1"
Sep 16 19:05:55.801477 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.801460    2580 scope.go:117] "RemoveContainer" containerID="18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1"
Sep 16 19:05:55.801733 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:05:55.801714    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1\": container with ID starting with 18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1 not found: ID does not exist" containerID="18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1"
Sep 16 19:05:55.801780 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.801739    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1"} err="failed to get container status \"18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1\": rpc error: code = NotFound desc = could not find container \"18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1\": container with ID starting with 18b0bec246226a241e25b9616f9a6b686fe697c3152c8056b36d1cb61c4581c1 not found: ID does not exist"
Sep 16 19:05:55.809809 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.809787    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"]
Sep 16 19:05:55.847418 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.847388    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e65cc4-976459b9-kcggk"]
Sep 16 19:05:55.848859 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.848845    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"]
Sep 16 19:05:55.850179 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:55.850159    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e65cc4-5cdfb9dff8-7ghvd"]
Sep 16 19:05:57.783060 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:57.783027    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a6375c0-59f8-4f29-864f-ceeaf950042f" path="/var/lib/kubelet/pods/8a6375c0-59f8-4f29-864f-ceeaf950042f/volumes"
Sep 16 19:05:57.783458 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:57.783396    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bce64fa3-476d-42a0-8b1f-997cdb57da4e" path="/var/lib/kubelet/pods/bce64fa3-476d-42a0-8b1f-997cdb57da4e/volumes"
Sep 16 19:05:57.841949 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:05:57.841918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:06:09.859206 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:06:09.859171    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:06:38.407983 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:06:38.407950    2580 scope.go:117] "RemoveContainer" containerID="33ada65f72d691c947ab10e91f52ac89ac8b0b05c2848c98b58be4726fe95438"
Sep 16 19:07:34.070966 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:07:34.070929    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:07:37.950785 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:07:37.950751    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 19:07:37.954118 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:07:37.954096    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 19:08:00.492771 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:00.492684    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:08:08.332533 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:08.332497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:08:15.773128 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:15.773080    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:08:22.748794 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:22.748755    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:08:40.646122 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:40.646086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:08:50.032087 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:50.032034    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:08:57.370200 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.370164    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"]
Sep 16 19:08:57.370650 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.370452    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc" podUID="92bc46d0-34a8-44aa-97d8-dd13c100d25d" containerName="manager" containerID="cri-o://c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787" gracePeriod=10
Sep 16 19:08:57.616488 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.616465    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 19:08:57.639873 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.639787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92bc46d0-34a8-44aa-97d8-dd13c100d25d-extensions-socket-volume\") pod \"92bc46d0-34a8-44aa-97d8-dd13c100d25d\" (UID: \"92bc46d0-34a8-44aa-97d8-dd13c100d25d\") "
Sep 16 19:08:57.639873 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.639822    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc9s7\" (UniqueName: \"kubernetes.io/projected/92bc46d0-34a8-44aa-97d8-dd13c100d25d-kube-api-access-lc9s7\") pod \"92bc46d0-34a8-44aa-97d8-dd13c100d25d\" (UID: \"92bc46d0-34a8-44aa-97d8-dd13c100d25d\") "
Sep 16 19:08:57.640198 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.640172    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bc46d0-34a8-44aa-97d8-dd13c100d25d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "92bc46d0-34a8-44aa-97d8-dd13c100d25d" (UID: "92bc46d0-34a8-44aa-97d8-dd13c100d25d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:57.642034 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.642011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92bc46d0-34a8-44aa-97d8-dd13c100d25d-kube-api-access-lc9s7" (OuterVolumeSpecName: "kube-api-access-lc9s7") pod "92bc46d0-34a8-44aa-97d8-dd13c100d25d" (UID: "92bc46d0-34a8-44aa-97d8-dd13c100d25d"). InnerVolumeSpecName "kube-api-access-lc9s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:57.670091 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.670056    2580 generic.go:356] "Generic (PLEG): container finished" podID="92bc46d0-34a8-44aa-97d8-dd13c100d25d" containerID="c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787" exitCode=0
Sep 16 19:08:57.670252 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.670118    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"
Sep 16 19:08:57.670252 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.670150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc" event={"ID":"92bc46d0-34a8-44aa-97d8-dd13c100d25d","Type":"ContainerDied","Data":"c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787"}
Sep 16 19:08:57.670252 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.670218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc" event={"ID":"92bc46d0-34a8-44aa-97d8-dd13c100d25d","Type":"ContainerDied","Data":"cbd952005638e36208f0ab0caeec6e438255624667f77225ab253c9e8cf40b60"}
Sep 16 19:08:57.670252 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.670249    2580 scope.go:117] "RemoveContainer" containerID="c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787"
Sep 16 19:08:57.680686 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.680668    2580 scope.go:117] "RemoveContainer" containerID="6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c"
Sep 16 19:08:57.688664 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.688635    2580 scope.go:117] "RemoveContainer" containerID="c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787"
Sep 16 19:08:57.688986 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:08:57.688964    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787\": container with ID starting with c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787 not found: ID does not exist" containerID="c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787"
Sep 16 19:08:57.689073 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.689000    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787"} err="failed to get container status \"c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787\": rpc error: code = NotFound desc = could not find container \"c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787\": container with ID starting with c0fecb57f5e7be35ce59fbfd96355b6411b6123b7455eb6a8cde8248c4ae6787 not found: ID does not exist"
Sep 16 19:08:57.689073 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.689025    2580 scope.go:117] "RemoveContainer" containerID="6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c"
Sep 16 19:08:57.689345 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:08:57.689312    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c\": container with ID starting with 6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c not found: ID does not exist" containerID="6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c"
Sep 16 19:08:57.689411 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.689349    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c"} err="failed to get container status \"6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c\": rpc error: code = NotFound desc = could not find container \"6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c\": container with ID starting with 6367e87ca1b7c22908d55da8dce11d2dc444a89c99886b80b0a7f45b702ac16c not found: ID does not exist"
Sep 16 19:08:57.699493 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.699468    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"]
Sep 16 19:08:57.704370 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.704347    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-997bc"]
Sep 16 19:08:57.740838 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.740806    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92bc46d0-34a8-44aa-97d8-dd13c100d25d-extensions-socket-volume\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:08:57.740838 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.740837    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc9s7\" (UniqueName: \"kubernetes.io/projected/92bc46d0-34a8-44aa-97d8-dd13c100d25d-kube-api-access-lc9s7\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:08:57.783812 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:08:57.783782    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92bc46d0-34a8-44aa-97d8-dd13c100d25d" path="/var/lib/kubelet/pods/92bc46d0-34a8-44aa-97d8-dd13c100d25d/volumes"
Sep 16 19:09:38.446104 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:09:38.446075    2580 scope.go:117] "RemoveContainer" containerID="a5150d8c654f71e5df9c92b95fe9dd564ed7911e44cb49618fe19ef7b1247e25"
Sep 16 19:10:20.527066 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:20.527031    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:10:27.471678 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:27.471635    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:10:35.429359 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:35.429310    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:10:38.467234 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:38.467198    2580 scope.go:117] "RemoveContainer" containerID="db07073a03a0b7e518a4b0a9723f22c38511e6fa9b0298a4264a7225a64d8ac1"
Sep 16 19:10:53.267392 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267359    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"]
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267681    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce64fa3-476d-42a0-8b1f-997cdb57da4e" containerName="payload-processing"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267692    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce64fa3-476d-42a0-8b1f-997cdb57da4e" containerName="payload-processing"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267709    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92bc46d0-34a8-44aa-97d8-dd13c100d25d" containerName="manager"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267714    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92bc46d0-34a8-44aa-97d8-dd13c100d25d" containerName="manager"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267730    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29af6dd-da7a-49e4-addd-35931065db84" containerName="revoke-keys"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267735    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29af6dd-da7a-49e4-addd-35931065db84" containerName="revoke-keys"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267750    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6375c0-59f8-4f29-864f-ceeaf950042f" containerName="payload-pre-processing"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267756    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6375c0-59f8-4f29-864f-ceeaf950042f" containerName="payload-pre-processing"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267762    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92bc46d0-34a8-44aa-97d8-dd13c100d25d" containerName="manager"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267766    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92bc46d0-34a8-44aa-97d8-dd13c100d25d" containerName="manager"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267812    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92bc46d0-34a8-44aa-97d8-dd13c100d25d" containerName="manager"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267819    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29af6dd-da7a-49e4-addd-35931065db84" containerName="revoke-keys"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267827    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce64fa3-476d-42a0-8b1f-997cdb57da4e" containerName="payload-processing"
Sep 16 19:10:53.267862 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.267833    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6375c0-59f8-4f29-864f-ceeaf950042f" containerName="payload-pre-processing"
Sep 16 19:10:53.270514 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.270497    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.274617 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:10:53.274592    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-same-id-a-e51427de-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-141-16.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-141-16.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"e2e-same-id-a-e51427de-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 16 19:10:53.301159 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.301131    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"]
Sep 16 19:10:53.327266 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.327239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-model-cache\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.327416 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.327277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tmp-dir\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.327416 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.327297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxt8\" (UniqueName: \"kubernetes.io/projected/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-kube-api-access-wqxt8\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.327548 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.327529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tls-certs\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.327585 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.327563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-dshm\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.327585 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.327579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-home\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.428703 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.428665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tls-certs\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.428703 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.428706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-dshm\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.428926 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.428723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-home\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.428926 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.428795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-model-cache\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.428926 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.428828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tmp-dir\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.428926 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.428860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxt8\" (UniqueName: \"kubernetes.io/projected/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-kube-api-access-wqxt8\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.429097 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.429077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-home\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.429224 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.429202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-model-cache\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.429265 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.429226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tmp-dir\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.430933 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.430912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-dshm\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:53.455265 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:53.455232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxt8\" (UniqueName: \"kubernetes.io/projected/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-kube-api-access-wqxt8\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:54.383968 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:54.383938    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e51427de-kserve-self-signed-certs\""
Sep 16 19:10:54.391276 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:54.391249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tls-certs\") pod \"e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") " pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:54.479792 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:54.479755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:10:54.767582 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:54.767552    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"]
Sep 16 19:10:54.770861 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:10:54.770830    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52b86e2e_a29a_4724_bd4a_c8f949a97d2b.slice/crio-734481fd4be4694879d29cf97b92cb477b963cda83be77e18f9cb53d92e94f21 WatchSource:0}: Error finding container 734481fd4be4694879d29cf97b92cb477b963cda83be77e18f9cb53d92e94f21: Status 404 returned error can't find the container with id 734481fd4be4694879d29cf97b92cb477b963cda83be77e18f9cb53d92e94f21
Sep 16 19:10:55.221074 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:55.220981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt" event={"ID":"52b86e2e-a29a-4724-bd4a-c8f949a97d2b","Type":"ContainerStarted","Data":"b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860"}
Sep 16 19:10:55.221074 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:55.221028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt" event={"ID":"52b86e2e-a29a-4724-bd4a-c8f949a97d2b","Type":"ContainerStarted","Data":"734481fd4be4694879d29cf97b92cb477b963cda83be77e18f9cb53d92e94f21"}
Sep 16 19:10:55.259198 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:55.259144    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt" podStartSLOduration=2.259127069 podStartE2EDuration="2.259127069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:55.257265613 +0000 UTC m=+2597.966505004" watchObservedRunningTime="2026-09-16 19:10:55.259127069 +0000 UTC m=+2597.968366454"
Sep 16 19:10:55.827399 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:10:55.827363    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:11:03.492524 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.492444    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"]
Sep 16 19:11:03.492891 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.492854    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92bc46d0-34a8-44aa-97d8-dd13c100d25d" containerName="manager"
Sep 16 19:11:03.495899 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.495884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.502303 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.502284    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e51427de-kserve-self-signed-certs\""
Sep 16 19:11:03.554081 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.554056    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"]
Sep 16 19:11:03.621587 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.621558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063401c7-7467-427d-9761-1b29dd48e129-tls-certs\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.621712 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.621593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-model-cache\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.621712 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.621613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-dshm\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.621784 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.621711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-tmp-dir\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.621784 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.621744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-home\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.621784 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.621765    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4j5t\" (UniqueName: \"kubernetes.io/projected/063401c7-7467-427d-9761-1b29dd48e129-kube-api-access-l4j5t\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.723025 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.722992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-tmp-dir\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.723025 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.723028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-home\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.723244 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.723048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4j5t\" (UniqueName: \"kubernetes.io/projected/063401c7-7467-427d-9761-1b29dd48e129-kube-api-access-l4j5t\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.723244 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.723098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063401c7-7467-427d-9761-1b29dd48e129-tls-certs\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.723244 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.723120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-model-cache\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.723244 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.723136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-dshm\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.723512 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.723488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-home\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.723512 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.723504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-tmp-dir\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.723621 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.723558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-model-cache\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.725403 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.725383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-dshm\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.725580 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.725563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063401c7-7467-427d-9761-1b29dd48e129-tls-certs\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.739591 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.739565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4j5t\" (UniqueName: \"kubernetes.io/projected/063401c7-7467-427d-9761-1b29dd48e129-kube-api-access-l4j5t\") pod \"e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") " pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:03.804748 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:03.804718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:04.127354 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:04.127318    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"]
Sep 16 19:11:04.129092 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:11:04.129058    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod063401c7_7467_427d_9761_1b29dd48e129.slice/crio-470d8c2e103f519765030f3bcf9b5dd6d192a7f548c6e50a3a7fd61d53dd8a1f WatchSource:0}: Error finding container 470d8c2e103f519765030f3bcf9b5dd6d192a7f548c6e50a3a7fd61d53dd8a1f: Status 404 returned error can't find the container with id 470d8c2e103f519765030f3bcf9b5dd6d192a7f548c6e50a3a7fd61d53dd8a1f
Sep 16 19:11:04.270248 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:04.270209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v" event={"ID":"063401c7-7467-427d-9761-1b29dd48e129","Type":"ContainerStarted","Data":"bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a"}
Sep 16 19:11:04.270248 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:04.270255    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v" event={"ID":"063401c7-7467-427d-9761-1b29dd48e129","Type":"ContainerStarted","Data":"470d8c2e103f519765030f3bcf9b5dd6d192a7f548c6e50a3a7fd61d53dd8a1f"}
Sep 16 19:11:04.310810 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:04.310763    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v" podStartSLOduration=1.310748831 podStartE2EDuration="1.310748831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:04.306236233 +0000 UTC m=+2607.015475624" watchObservedRunningTime="2026-09-16 19:11:04.310748831 +0000 UTC m=+2607.019988223"
Sep 16 19:11:04.480169 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:04.480062    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:11:04.480169 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:04.480115    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:11:04.492664 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:04.492641    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:11:05.286142 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:05.286111    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:11:05.775685 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:05.775648    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:11:13.805010 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:13.804976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:13.805589 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:13.805037    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:13.817536 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:13.817517    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:14.330136 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:14.330100    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:38.486255 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:38.486221    2580 scope.go:117] "RemoveContainer" containerID="88116a47180f381efa7f184f3e74eb6cfc20e3e38fbb91dbdbbfc347146367c0"
Sep 16 19:11:38.494455 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:38.494432    2580 scope.go:117] "RemoveContainer" containerID="0f130fe42b8656a1a7e49f5dbc7fa028444abc7cfe4b29a914b5822a37b14575"
Sep 16 19:11:40.990157 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:40.990119    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"]
Sep 16 19:11:43.521613 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:43.521577    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:11:43.672923 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:43.672891    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"]
Sep 16 19:11:53.226182 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:53.226151    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:11:56.009188 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.009143    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt" podUID="52b86e2e-a29a-4724-bd4a-c8f949a97d2b" containerName="main" containerID="cri-o://b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860" gracePeriod=45
Sep 16 19:11:56.256803 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.256780    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:11:56.412690 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.412641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-model-cache\") pod \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") "
Sep 16 19:11:56.412903 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.412738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqxt8\" (UniqueName: \"kubernetes.io/projected/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-kube-api-access-wqxt8\") pod \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") "
Sep 16 19:11:56.412903 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.412778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tmp-dir\") pod \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") "
Sep 16 19:11:56.412903 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.412808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tls-certs\") pod \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") "
Sep 16 19:11:56.412903 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.412837    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-home\") pod \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") "
Sep 16 19:11:56.413138 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.412942    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-dshm\") pod \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\" (UID: \"52b86e2e-a29a-4724-bd4a-c8f949a97d2b\") "
Sep 16 19:11:56.413138 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.412970    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-model-cache" (OuterVolumeSpecName: "model-cache") pod "52b86e2e-a29a-4724-bd4a-c8f949a97d2b" (UID: "52b86e2e-a29a-4724-bd4a-c8f949a97d2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:56.413138 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.413076    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52b86e2e-a29a-4724-bd4a-c8f949a97d2b" (UID: "52b86e2e-a29a-4724-bd4a-c8f949a97d2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:56.413273 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.413170    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-home" (OuterVolumeSpecName: "home") pod "52b86e2e-a29a-4724-bd4a-c8f949a97d2b" (UID: "52b86e2e-a29a-4724-bd4a-c8f949a97d2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:56.413273 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.413233    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tmp-dir\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:56.413273 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.413253    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-model-cache\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:56.415048 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.415026    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-dshm" (OuterVolumeSpecName: "dshm") pod "52b86e2e-a29a-4724-bd4a-c8f949a97d2b" (UID: "52b86e2e-a29a-4724-bd4a-c8f949a97d2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:56.415124 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.415070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-kube-api-access-wqxt8" (OuterVolumeSpecName: "kube-api-access-wqxt8") pod "52b86e2e-a29a-4724-bd4a-c8f949a97d2b" (UID: "52b86e2e-a29a-4724-bd4a-c8f949a97d2b"). InnerVolumeSpecName "kube-api-access-wqxt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:56.415162 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.415134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52b86e2e-a29a-4724-bd4a-c8f949a97d2b" (UID: "52b86e2e-a29a-4724-bd4a-c8f949a97d2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:56.514418 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.514381    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-home\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:56.514418 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.514410    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-dshm\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:56.514418 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.514423    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqxt8\" (UniqueName: \"kubernetes.io/projected/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-kube-api-access-wqxt8\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:56.514657 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.514436    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b86e2e-a29a-4724-bd4a-c8f949a97d2b-tls-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:56.522181 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.522150    2580 generic.go:356] "Generic (PLEG): container finished" podID="52b86e2e-a29a-4724-bd4a-c8f949a97d2b" containerID="b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860" exitCode=0
Sep 16 19:11:56.522309 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.522189    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"
Sep 16 19:11:56.522309 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.522227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt" event={"ID":"52b86e2e-a29a-4724-bd4a-c8f949a97d2b","Type":"ContainerDied","Data":"b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860"}
Sep 16 19:11:56.522309 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.522262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt" event={"ID":"52b86e2e-a29a-4724-bd4a-c8f949a97d2b","Type":"ContainerDied","Data":"734481fd4be4694879d29cf97b92cb477b963cda83be77e18f9cb53d92e94f21"}
Sep 16 19:11:56.522309 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.522281    2580 scope.go:117] "RemoveContainer" containerID="b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860"
Sep 16 19:11:56.530899 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.530880    2580 scope.go:117] "RemoveContainer" containerID="b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860"
Sep 16 19:11:56.531179 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:11:56.531160    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860\": container with ID starting with b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860 not found: ID does not exist" containerID="b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860"
Sep 16 19:11:56.531247 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.531187    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860"} err="failed to get container status \"b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860\": rpc error: code = NotFound desc = could not find container \"b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860\": container with ID starting with b7001dec8ddc0d4ffa4f8eaf12cb4e013f603625414c62dce41fa15f3cb6a860 not found: ID does not exist"
Sep 16 19:11:56.558638 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.558595    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"]
Sep 16 19:11:56.568980 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:56.568948    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e51427de-kserve-6b94747df5-5l4tt"]
Sep 16 19:11:57.784270 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:57.784239    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52b86e2e-a29a-4724-bd4a-c8f949a97d2b" path="/var/lib/kubelet/pods/52b86e2e-a29a-4724-bd4a-c8f949a97d2b/volumes"
Sep 16 19:11:58.691405 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:58.691369    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v" podUID="063401c7-7467-427d-9761-1b29dd48e129" containerName="main" containerID="cri-o://bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a" gracePeriod=45
Sep 16 19:11:58.946508 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:58.946454    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:59.137619 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.137576    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-tmp-dir\") pod \"063401c7-7467-427d-9761-1b29dd48e129\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") "
Sep 16 19:11:59.137795 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.137633    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4j5t\" (UniqueName: \"kubernetes.io/projected/063401c7-7467-427d-9761-1b29dd48e129-kube-api-access-l4j5t\") pod \"063401c7-7467-427d-9761-1b29dd48e129\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") "
Sep 16 19:11:59.137795 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.137710    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-home\") pod \"063401c7-7467-427d-9761-1b29dd48e129\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") "
Sep 16 19:11:59.137795 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.137728    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063401c7-7467-427d-9761-1b29dd48e129-tls-certs\") pod \"063401c7-7467-427d-9761-1b29dd48e129\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") "
Sep 16 19:11:59.137964 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.137837    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "063401c7-7467-427d-9761-1b29dd48e129" (UID: "063401c7-7467-427d-9761-1b29dd48e129"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:59.137964 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.137843    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-dshm\") pod \"063401c7-7467-427d-9761-1b29dd48e129\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") "
Sep 16 19:11:59.137964 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.137903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-model-cache\") pod \"063401c7-7467-427d-9761-1b29dd48e129\" (UID: \"063401c7-7467-427d-9761-1b29dd48e129\") "
Sep 16 19:11:59.138130 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.138033    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-home" (OuterVolumeSpecName: "home") pod "063401c7-7467-427d-9761-1b29dd48e129" (UID: "063401c7-7467-427d-9761-1b29dd48e129"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:59.138210 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.138186    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-model-cache" (OuterVolumeSpecName: "model-cache") pod "063401c7-7467-427d-9761-1b29dd48e129" (UID: "063401c7-7467-427d-9761-1b29dd48e129"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:59.138270 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.138255    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-home\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:59.138320 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.138278    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-tmp-dir\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:59.139803 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.139781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/063401c7-7467-427d-9761-1b29dd48e129-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "063401c7-7467-427d-9761-1b29dd48e129" (UID: "063401c7-7467-427d-9761-1b29dd48e129"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:59.139901 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.139814    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-dshm" (OuterVolumeSpecName: "dshm") pod "063401c7-7467-427d-9761-1b29dd48e129" (UID: "063401c7-7467-427d-9761-1b29dd48e129"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:59.139901 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.139817    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063401c7-7467-427d-9761-1b29dd48e129-kube-api-access-l4j5t" (OuterVolumeSpecName: "kube-api-access-l4j5t") pod "063401c7-7467-427d-9761-1b29dd48e129" (UID: "063401c7-7467-427d-9761-1b29dd48e129"). InnerVolumeSpecName "kube-api-access-l4j5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:59.238680 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.238625    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4j5t\" (UniqueName: \"kubernetes.io/projected/063401c7-7467-427d-9761-1b29dd48e129-kube-api-access-l4j5t\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:59.238680 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.238645    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063401c7-7467-427d-9761-1b29dd48e129-tls-certs\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:59.238680 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.238656    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-dshm\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:59.238680 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.238664    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063401c7-7467-427d-9761-1b29dd48e129-model-cache\") on node \"ip-10-0-141-16.ec2.internal\" DevicePath \"\""
Sep 16 19:11:59.539109 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.539071    2580 generic.go:356] "Generic (PLEG): container finished" podID="063401c7-7467-427d-9761-1b29dd48e129" containerID="bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a" exitCode=0
Sep 16 19:11:59.539278 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.539126    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"
Sep 16 19:11:59.539278 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.539143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v" event={"ID":"063401c7-7467-427d-9761-1b29dd48e129","Type":"ContainerDied","Data":"bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a"}
Sep 16 19:11:59.539278 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.539176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v" event={"ID":"063401c7-7467-427d-9761-1b29dd48e129","Type":"ContainerDied","Data":"470d8c2e103f519765030f3bcf9b5dd6d192a7f548c6e50a3a7fd61d53dd8a1f"}
Sep 16 19:11:59.539278 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.539191    2580 scope.go:117] "RemoveContainer" containerID="bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a"
Sep 16 19:11:59.548698 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.548679    2580 scope.go:117] "RemoveContainer" containerID="bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a"
Sep 16 19:11:59.548959 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:11:59.548941    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a\": container with ID starting with bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a not found: ID does not exist" containerID="bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a"
Sep 16 19:11:59.549010 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.548968    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a"} err="failed to get container status \"bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a\": rpc error: code = NotFound desc = could not find container \"bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a\": container with ID starting with bde2ac4a860c0203f2230558a8510239c71c1282e5f187c44b6ef148a7463f0a not found: ID does not exist"
Sep 16 19:11:59.575422 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.575395    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"]
Sep 16 19:11:59.591686 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.591663    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e51427de-kserve-69d44bb9d7-vwm6v"]
Sep 16 19:11:59.783142 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:11:59.783113    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="063401c7-7467-427d-9761-1b29dd48e129" path="/var/lib/kubelet/pods/063401c7-7467-427d-9761-1b29dd48e129/volumes"
Sep 16 19:12:01.476786 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:01.476751    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:12:08.286138 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:08.286103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:12:15.587625 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:15.587591    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:12:24.394531 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:24.394493    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:12:30.594240 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:30.594156    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:12:37.200953 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:37.200922    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v9l7j"]
Sep 16 19:12:37.978890 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:37.978862    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 19:12:37.983840 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:37.983813    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 19:12:38.515282 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:38.515254    2580 scope.go:117] "RemoveContainer" containerID="f06c16215cac4594cfb7bf8b03866da47f2a18407b8b6ba2311e1c60228c7fc8"
Sep 16 19:12:40.173609 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:40.173575    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b47bf9dd7-dsq4b_05469e3c-de98-43ac-a7a4-d8eac0bd013b/authorino/0.log"
Sep 16 19:12:46.776169 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:46.776139    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-744584488c-2gtqt_61e714c0-5317-47c8-ae86-623a63f55a5d/copy-manifests/0.log"
Sep 16 19:12:46.809735 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:46.809710    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-744584488c-2gtqt_61e714c0-5317-47c8-ae86-623a63f55a5d/manager/0.log"
Sep 16 19:12:47.851735 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:47.851703    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826405-8lcv5_224064a5-060e-4ff9-bf02-cc1b10971d35/cleanup/0.log"
Sep 16 19:12:47.973067 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:47.973030    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826420-67hp6_7edb2656-2aa2-4b54-8191-6c9849cecc0d/cleanup/0.log"
Sep 16 19:12:48.914620 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:48.914577    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b47bf9dd7-dsq4b_05469e3c-de98-43ac-a7a4-d8eac0bd013b/authorino/0.log"
Sep 16 19:12:49.735199 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:49.735167    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-v9l7j_5fffb0aa-a7bd-4c8c-8660-bd3079b73181/limitador/0.log"
Sep 16 19:12:50.720807 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:50.720765    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbdbf5bc-mwsnr_f923cd31-3318-4e76-8c1a-5c515f868aa6/kube-auth-proxy/0.log"
Sep 16 19:12:52.041888 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:52.041861    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-zhk7w_f2a7cb9f-9714-4467-bd76-9ce0b9b23733/main/0.log"
Sep 16 19:12:52.304715 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:52.304639    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsv9rn_45e0ca04-996d-4c0f-a928-312369224450/main/0.log"
Sep 16 19:12:52.583727 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:52.583648    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-tgcrw_7b874b7a-cfb2-4d8c-8ece-a804cd4be3c0/main/0.log"
Sep 16 19:12:58.301160 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.301126    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-km8mc/must-gather-pwjst"]
Sep 16 19:12:58.301577 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.301476    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b86e2e-a29a-4724-bd4a-c8f949a97d2b" containerName="main"
Sep 16 19:12:58.301577 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.301487    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b86e2e-a29a-4724-bd4a-c8f949a97d2b" containerName="main"
Sep 16 19:12:58.301577 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.301499    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063401c7-7467-427d-9761-1b29dd48e129" containerName="main"
Sep 16 19:12:58.301577 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.301507    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="063401c7-7467-427d-9761-1b29dd48e129" containerName="main"
Sep 16 19:12:58.301577 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.301561    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="063401c7-7467-427d-9761-1b29dd48e129" containerName="main"
Sep 16 19:12:58.301577 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.301570    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="52b86e2e-a29a-4724-bd4a-c8f949a97d2b" containerName="main"
Sep 16 19:12:58.304554 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.304529    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-km8mc/must-gather-pwjst"
Sep 16 19:12:58.316368 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.316347    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-km8mc\"/\"openshift-service-ca.crt\""
Sep 16 19:12:58.316474 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.316374    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-km8mc\"/\"kube-root-ca.crt\""
Sep 16 19:12:58.326125 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.326102    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-km8mc/must-gather-pwjst"]
Sep 16 19:12:58.343778 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.343755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rngh\" (UniqueName: \"kubernetes.io/projected/80101487-a3e2-4a4c-a471-2b65db21f901-kube-api-access-6rngh\") pod \"must-gather-pwjst\" (UID: \"80101487-a3e2-4a4c-a471-2b65db21f901\") " pod="openshift-must-gather-km8mc/must-gather-pwjst"
Sep 16 19:12:58.343893 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.343817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80101487-a3e2-4a4c-a471-2b65db21f901-must-gather-output\") pod \"must-gather-pwjst\" (UID: \"80101487-a3e2-4a4c-a471-2b65db21f901\") " pod="openshift-must-gather-km8mc/must-gather-pwjst"
Sep 16 19:12:58.444361 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.444300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rngh\" (UniqueName: \"kubernetes.io/projected/80101487-a3e2-4a4c-a471-2b65db21f901-kube-api-access-6rngh\") pod \"must-gather-pwjst\" (UID: \"80101487-a3e2-4a4c-a471-2b65db21f901\") " pod="openshift-must-gather-km8mc/must-gather-pwjst"
Sep 16 19:12:58.444540 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.444411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80101487-a3e2-4a4c-a471-2b65db21f901-must-gather-output\") pod \"must-gather-pwjst\" (UID: \"80101487-a3e2-4a4c-a471-2b65db21f901\") " pod="openshift-must-gather-km8mc/must-gather-pwjst"
Sep 16 19:12:58.444717 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.444701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80101487-a3e2-4a4c-a471-2b65db21f901-must-gather-output\") pod \"must-gather-pwjst\" (UID: \"80101487-a3e2-4a4c-a471-2b65db21f901\") " pod="openshift-must-gather-km8mc/must-gather-pwjst"
Sep 16 19:12:58.482668 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.482643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rngh\" (UniqueName: \"kubernetes.io/projected/80101487-a3e2-4a4c-a471-2b65db21f901-kube-api-access-6rngh\") pod \"must-gather-pwjst\" (UID: \"80101487-a3e2-4a4c-a471-2b65db21f901\") " pod="openshift-must-gather-km8mc/must-gather-pwjst"
Sep 16 19:12:58.612993 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.612907    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-km8mc/must-gather-pwjst"
Sep 16 19:12:58.913948 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.913923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-km8mc/must-gather-pwjst"]
Sep 16 19:12:58.916268 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:12:58.916237    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80101487_a3e2_4a4c_a471_2b65db21f901.slice/crio-2b358f71ddfef7736c7ce4693854158ced68c6a4473dea9e3e60ba13bf256b4b WatchSource:0}: Error finding container 2b358f71ddfef7736c7ce4693854158ced68c6a4473dea9e3e60ba13bf256b4b: Status 404 returned error can't find the container with id 2b358f71ddfef7736c7ce4693854158ced68c6a4473dea9e3e60ba13bf256b4b
Sep 16 19:12:58.918112 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:58.918097    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:12:59.817483 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:12:59.817448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-km8mc/must-gather-pwjst" event={"ID":"80101487-a3e2-4a4c-a471-2b65db21f901","Type":"ContainerStarted","Data":"2b358f71ddfef7736c7ce4693854158ced68c6a4473dea9e3e60ba13bf256b4b"}
Sep 16 19:13:00.824505 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:00.824469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-km8mc/must-gather-pwjst" event={"ID":"80101487-a3e2-4a4c-a471-2b65db21f901","Type":"ContainerStarted","Data":"f1e4fbff6c7974a9193368ca138f43f14bed56a76405d5a90b181d18d4a1a435"}
Sep 16 19:13:00.825068 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:00.825046    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-km8mc/must-gather-pwjst" event={"ID":"80101487-a3e2-4a4c-a471-2b65db21f901","Type":"ContainerStarted","Data":"6bf94dea5f79030ddfa169998257cb060b48d67a1b13b6490321b1b98dca8ee9"}
Sep 16 19:13:03.235083 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:03.235046    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-txjjd_be53619d-4600-4740-b307-ab98564ff088/global-pull-secret-syncer/0.log"
Sep 16 19:13:03.545695 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:03.545660    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mdgrv_eb761fe6-5e93-438b-a415-2c50316fc733/konnectivity-agent/0.log"
Sep 16 19:13:04.017033 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:04.016992    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-16.ec2.internal_6f3efc08d1fe8fd9d7d17f4d752bb79b/haproxy/0.log"
Sep 16 19:13:06.712597 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:06.712497    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b47bf9dd7-dsq4b_05469e3c-de98-43ac-a7a4-d8eac0bd013b/authorino/0.log"
Sep 16 19:13:07.152443 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:07.152412    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-v9l7j_5fffb0aa-a7bd-4c8c-8660-bd3079b73181/limitador/0.log"
Sep 16 19:13:10.449765 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:10.449728    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-857fb664c9-fd5kt_4acb3796-9cdf-408e-8b33-125d56846404/metrics-server/0.log"
Sep 16 19:13:10.591095 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:10.591057    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8qg9_016d2d6f-9c6f-4987-8d87-0aa89fdc568f/node-exporter/0.log"
Sep 16 19:13:10.674830 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:10.674794    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8qg9_016d2d6f-9c6f-4987-8d87-0aa89fdc568f/kube-rbac-proxy/0.log"
Sep 16 19:13:10.757082 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:10.756971    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8qg9_016d2d6f-9c6f-4987-8d87-0aa89fdc568f/init-textfile/0.log"
Sep 16 19:13:11.326594 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.326533    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-km8mc/must-gather-pwjst" podStartSLOduration=12.387527133 podStartE2EDuration="13.326515156s" totalImagesPullingTime="938.988023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:00.894873934 +0000 UTC m=+2723.604113327" watchObservedRunningTime="2026-09-16 19:13:11.326515156 +0000 UTC m=+2734.035754547"
Sep 16 19:13:11.327777 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.327747    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"]
Sep 16 19:13:11.331399 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.331376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.335827 ip-10-0-141-16 kubenswrapper[2580]: E0916 19:13:11.335796    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"perf-node-gather-daemonset-69nnx\" is forbidden: User \"system:node:ip-10-0-141-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-km8mc\": no relationship found between node 'ip-10-0-141-16.ec2.internal' and this object" podUID="9075877e-5dce-4f9d-9438-2a6db2fd14ea" pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.340581 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.340553    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-km8mc\"/\"default-dockercfg-5tbwq\""
Sep 16 19:13:11.350628 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.350595    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"]
Sep 16 19:13:11.378138 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.378104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngp2\" (UniqueName: \"kubernetes.io/projected/9075877e-5dce-4f9d-9438-2a6db2fd14ea-kube-api-access-zngp2\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.378491 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.378468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-sys\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.378642 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.378628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-podres\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.378829 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.378815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-lib-modules\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.378960 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.378948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-proc\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.479986 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.479940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngp2\" (UniqueName: \"kubernetes.io/projected/9075877e-5dce-4f9d-9438-2a6db2fd14ea-kube-api-access-zngp2\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.480502 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.480010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-sys\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.480502 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.480045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-podres\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.480502 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.480109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-lib-modules\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.480502 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.480140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-sys\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.480502 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.480211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-proc\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.480502 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.480150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-proc\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.480502 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.480277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-podres\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.480502 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.480395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9075877e-5dce-4f9d-9438-2a6db2fd14ea-lib-modules\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.494735 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.494705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngp2\" (UniqueName: \"kubernetes.io/projected/9075877e-5dce-4f9d-9438-2a6db2fd14ea-kube-api-access-zngp2\") pod \"perf-node-gather-daemonset-69nnx\" (UID: \"9075877e-5dce-4f9d-9438-2a6db2fd14ea\") " pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.587215 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.587135    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_367c7793-f1db-4a24-a898-ce6a170d79dc/prometheus/0.log"
Sep 16 19:13:11.643712 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.643666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:11.679950 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.679917    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_367c7793-f1db-4a24-a898-ce6a170d79dc/config-reloader/0.log"
Sep 16 19:13:11.762114 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.762034    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_367c7793-f1db-4a24-a898-ce6a170d79dc/thanos-sidecar/0.log"
Sep 16 19:13:11.831880 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.831845    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_367c7793-f1db-4a24-a898-ce6a170d79dc/kube-rbac-proxy-web/0.log"
Sep 16 19:13:11.900383 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.900265    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_367c7793-f1db-4a24-a898-ce6a170d79dc/kube-rbac-proxy/0.log"
Sep 16 19:13:11.943347 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.938921    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"]
Sep 16 19:13:11.949258 ip-10-0-141-16 kubenswrapper[2580]: W0916 19:13:11.949212    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9075877e_5dce_4f9d_9438_2a6db2fd14ea.slice/crio-a55ab96b002897e5aaaacfe83728db2de8c8461d2dc3ae05fe9c43b8da674ee3 WatchSource:0}: Error finding container a55ab96b002897e5aaaacfe83728db2de8c8461d2dc3ae05fe9c43b8da674ee3: Status 404 returned error can't find the container with id a55ab96b002897e5aaaacfe83728db2de8c8461d2dc3ae05fe9c43b8da674ee3
Sep 16 19:13:11.982604 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:11.982584    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_367c7793-f1db-4a24-a898-ce6a170d79dc/kube-rbac-proxy-thanos/0.log"
Sep 16 19:13:12.049855 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:12.049823    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_367c7793-f1db-4a24-a898-ce6a170d79dc/init-config-reloader/0.log"
Sep 16 19:13:12.132450 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:12.132414    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-6j8xp_c8009d09-30df-4e68-9b01-a7b39cc41763/prometheus-operator/0.log"
Sep 16 19:13:12.194999 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:12.194898    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-6j8xp_c8009d09-30df-4e68-9b01-a7b39cc41763/kube-rbac-proxy/0.log"
Sep 16 19:13:12.269380 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:12.269302    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-h2vzf_4bfef989-8ac2-465a-aac5-eabd93813538/prometheus-operator-admission-webhook/0.log"
Sep 16 19:13:12.909407 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:12.909369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx" event={"ID":"9075877e-5dce-4f9d-9438-2a6db2fd14ea","Type":"ContainerStarted","Data":"0d7deeb902d3dc81a2b9863d9ab8db9803d4ee15c0a8b0048e6ce6692100f663"}
Sep 16 19:13:12.909824 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:12.909436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx" event={"ID":"9075877e-5dce-4f9d-9438-2a6db2fd14ea","Type":"ContainerStarted","Data":"a55ab96b002897e5aaaacfe83728db2de8c8461d2dc3ae05fe9c43b8da674ee3"}
Sep 16 19:13:12.909824 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:12.909505    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:12.965888 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:12.965813    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx" podStartSLOduration=1.965787111 podStartE2EDuration="1.965787111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:12.9650326 +0000 UTC m=+2735.674271993" watchObservedRunningTime="2026-09-16 19:13:12.965787111 +0000 UTC m=+2735.675026505"
Sep 16 19:13:14.228611 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:14.228584    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-2d65d_9b7e2fe2-787d-4a41-994b-d833daae8743/networking-console-plugin/0.log"
Sep 16 19:13:15.046187 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:15.046154    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/2.log"
Sep 16 19:13:15.049761 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:15.049741    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bbkt2_4222a382-cb85-4cd2-b8ed-0dd10fa26702/console-operator/3.log"
Sep 16 19:13:18.177916 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:18.177884    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8wtgj_ef3fcd16-81ea-4a0a-8b03-f63208709c03/dns/0.log"
Sep 16 19:13:18.220887 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:18.220859    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8wtgj_ef3fcd16-81ea-4a0a-8b03-f63208709c03/kube-rbac-proxy/0.log"
Sep 16 19:13:18.447210 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:18.447131    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r5z5r_7fdbe8e9-3ed9-49e4-a10d-41c1699e7a46/dns-node-resolver/0.log"
Sep 16 19:13:18.928488 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:18.928453    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-km8mc/perf-node-gather-daemonset-69nnx"
Sep 16 19:13:19.373741 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:19.373677    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bfn66_cdadcf08-0ff4-42ac-8ee4-3deb381bd5a2/node-ca/0.log"
Sep 16 19:13:21.695760 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:21.695672    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbdbf5bc-mwsnr_f923cd31-3318-4e76-8c1a-5c515f868aa6/kube-auth-proxy/0.log"
Sep 16 19:13:23.209138 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:23.209107    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbdbf5bc-mwsnr_f923cd31-3318-4e76-8c1a-5c515f868aa6/kube-auth-proxy/0.log"
Sep 16 19:13:23.323207 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:23.323180    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g8th6_810613e1-3990-43c3-b8f8-6bc0a1c2c81b/serve-healthcheck-canary/0.log"
Sep 16 19:13:24.189042 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:24.189015    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2jvx9_8017534a-2ecf-4a17-87cc-2395a913d221/kube-rbac-proxy/0.log"
Sep 16 19:13:24.301908 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:24.301876    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2jvx9_8017534a-2ecf-4a17-87cc-2395a913d221/exporter/0.log"
Sep 16 19:13:24.407267 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:24.407239    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2jvx9_8017534a-2ecf-4a17-87cc-2395a913d221/extractor/0.log"
Sep 16 19:13:26.060625 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:26.060594    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbdbf5bc-mwsnr_f923cd31-3318-4e76-8c1a-5c515f868aa6/kube-auth-proxy/0.log"
Sep 16 19:13:30.028438 ip-10-0-141-16 kubenswrapper[2580]: I0916 19:13:30.028403    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-744584488c-2gtqt_61e714c0-5317-47c8-ae86-623a63f55a5d/manager/0.log"